Wearable electronic, multi-sensory, human/machine, human/human interfaces

ABSTRACT

A wearable Haptic Humaxi/Machine Interface (HHMI) receives electrical activity from muscles and nerves of a user. An electrical signal is determined having characteristics based on the received electrical activity. The electrical signal is generated and applied to an object to cause an action dependent on the received electrical activity. The object can be a biological component of the user, such as a muscle, another user, or a remotely located machine such as a drone. Exemplary uses include mitigating tremor, accelerated learning, cognitive therapy, remote robotic, drone and probe control and sensing, virtual and augmented reality, stroke, brain and spinal cord rehabilitation, gaming, education, pain relief, entertainment, remote surgery, remote participation in and/or observation of an event such as a sporting event, biofeedback and remotality. Remotality is the perception of a reality occurring remote from the user. The reality may be remote in time, location and/or physical form. The reality may be consistent with the natural world, comprised of an alternative, fictional world or a mixture of natural and fictional constituents.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is the US National Stage Utility Patent Application ofthe corresponding PCT Application No. PCT/US2016/026930, InternationalFiling Date Apr. 11, 2016, entitled Wearable Electronic, Multi-Sensory,Human/Machine, Human/Human Interfaces and claims priority of U.S.Provisional Patent Application No. 62/147,016, Filed Apr. 14, 2015,entitled Multi-Sensory Human/Machine, Human/Human Interfaces and U.S.Provisional Patent Application No. 62/253,767, filed Nov. 11, 2015,entitled Wearable Electronic Human/Machine Interface for MitigatingTremor, Accelerated Learning, Cognitive Therapy, Remote Control, andVirtual and Augmented Reality; and is related to U.S. Utility patentapplication Ser. No. 14/269,133, filed on May 3, 2014, entitledAccelerated Learning, Entertainment and Cognitive Therapy UsingAugmented Reality Comprising Haptic, Auditory, and Visual Stimulationwhich is the Utility application of U.S. Provisional Application No.61/818,971, filed on May 3, 2013, entitled Accelerated Learning,Entertainment and Cognitive Therapy Using Augmented Reality ComprisingHaptic, Auditory, and Visual Stimulation. These applications are allincorporated herein in their entirety.

TECHNICAL FIELD

The present invention relates to a remote reality (“remotality”)interface between humans and machines, and between humans and humans.More particularly, the present invention pertains to a wearable HapticHuman/Machine Interface (HHMI) for uses including, but not limited to,mitigating tremor, accelerated learning, cognitive therapy, remoterobotic, drone and probe control and sensing, virtual and augmentedreality, stroke, brain and spinal cord rehabilitation, gaming,education, pain relief, entertainment, remote surgery, remoteparticipation in and/or observation of an event such as a sportingevent, and biofeedback.

BACKGROUND OF THE INVENTION

This section is intended to provide a background or context to theinventions disclosed below. The description herein may include conceptsthat could be pursued, but are not necessarily ones that have beenpreviously conceived, implemented or described. Therefore, unlessotherwise explicitly indicated herein, what is described in this sectionis not prior art to the description in this application and is notadmitted to be prior art by inclusion in this section.

Virtual Reality may be defined as a computer-generated simulation of athree-dimensional image or environment that can be interacted with in aseemingly real or physical way by a user using special electronicequipment, such as goggles, headphones and gloves fitted with sensorycue transducers.

Augmented reality may be defined as a live, direct or indirect, view ofa physical, real-world environment whose elements are augmented bycomputer-generated sensory input such as sound, video, graphics or GPSdata.

Electromyography (EMG) is the recording of electrical activity in themuscles, typically through a surface transducer in communication withthe skin of a user. An evoked potential or evoked response is anelectrical potential obtained from the nervous system following astimulus. Spontaneous potentials may be detected byelectroencephalography (EEG), electromyography (EMG), or otherelectrophysiological recording methods. An event-related potential (ERP)is the measured brain response that is the direct result of a specificsensory, cognitive, or motor event.

Electroencephalography (EEG) is the recording of electrical activityalong the scalp. Electrocardiogram (EKG) is the recording of electricaland muscle activity of the heart. Electromyography (EMG) is therecording of electrical activity in the muscles, typically through asurface transducer in communication with the skin of a user. An evokedpotential or evoked response is an electrical potential obtained fromthe nervous system following a stimulus. Spontaneous potentials asdetected by electroencephalography (EEG), electromyography (EMG), orother electrophysiological recording methods. An event-related potential(ERP) is the measured brain response that is the direct result of aspecific sensory, cognitive, or motor event.

The use of electrical stimulation for pain relief and muscular trainingis known, for example, as transcutaneous electrical nerve stimulation(TENS) and can be used to stimulate nerve endings to block pain.Neuromuscular electrical stimulation (NMES) has been used for causinginvoluntary contractions to build and tone muscles for sports, fitnessand rehabilitation.

In the human body, somatic and kinesthetic sensations relate to forceand touch. Somatic sensations, for example, are perceived cutaneous (atthe skin) and subcutaneous (below the skin). Kinesthetic sensations aremore related to mechanical body parts, such as joints and muscles. Ingeneral, these sensations can be called haptic feedback which can beused to determine things like geometry, roughness, slipperiness,temperature, weight and inertia (force).

SUMMARY OF THE INVENTION

In accordance with a non-limiting exemplary embodiment, time sequentialdata is received from a remote transmitter. Time sequential data isdefined herein as time series data or information that is generated,recorded, transmitted and/or received as an analog signal and/or as adigital signal, where the data or information varies over time such asin the case of a video signal or audio signal of an event. A pluralityof haptic sensory cues are generated capable of being perceived by auser. The haptic sensory cues are received by the user as computercontrolled serially generated electrical signals. The electrical signalsinvoke a perception by the user related to the sense of touch. Thehaptic sensory cues are generated in synchronization dependent on thetime sequential data.

In accordance with another non-limiting embodiment, a plurality of firstsensory cues are generated that are capable of being perceived by auser. The plurality of first sensory cues are time sequentiallygenerated and effective for stimulating at least one sense of the user.A plurality of haptic sensory cues are generated capable of beingperceived by the user. The haptic sensory cues are received by the useras computer controlled time sequentially generated electrical signals.The electrical signals invoke a perception by the user related to thesense of touch. The haptic sensory cues are generated in synchronizationdependent on the time sequentially generated plurality of first sensorycues.

In accordance with another non-limiting exemplary embodiment, ahuman/machine interface includes a plurality of conductive patches forapplying an electrical signal through the skin of a user to stimulateelectrical signal receptors. A signal generator generates a plurality ofhaptic cues in the form of electrical signals applied to the skin of theuser through the plurality of conductive patches. The plurality ofhaptic sensory cues are capable of being perceived by a sense of touchor muscle movement of the user.

In accordance with another non-limiting exemplary embodiment, aplurality of haptic sensory cues are generated capable of beingperceived by a user. The plurality of haptic sensory cues are dependenton a determined condition of at least one movable member of a performingbody performing an event. The plurality of haptic sensory cues areeffective for stimulating a touch processing center of a brain of theuser based on the determined condition. A plurality of visual sensorycues are generated capable of being displayed to the user on a videodisplay device. The visual sensory cues provide a virtual visualindication to the user of a position of at least one of the at least onemoveable member and the performing body. The visual sensory cues areeffective for stimulating the visual processing center of the brain ofthe user. The visual sensory cues are synchronized with the hapticsensory cues so that the position is virtually visually indicated insynchronization with the haptic sensory cues, and so that the visualprocessing center is stimulated with the visual sensory cues insynchronization with the haptic sensory cues stimulating the touchprocessing center.

Remotality, in accordance with the inventive aspects described herein,is the perception of a reality occurring remote from the user. Thereality may be remote in time, location and/or physical form. Thereality may be consistent with the natural world, comprised of analternative, fictional world or a mixture of natural and fictionalconstituents. In accordance with exemplary embodiments, the immersion ofthe user into the remotality experience may be as complete as practical,such as is the goal of some virtual reality systems, or it may be apartial immersion, such as is the goal of some augmented realitysystems.

In accordance with an aspect of the invention, a plurality of hapticsensory cues are generated capable of being perceived by a user. Thehaptic sensory cues are received by the user as computer controlledserially generated electrical signals. The electrical signals invoke atleast one of an involuntary body part movement and a perception by theuser. The involuntary body part movement causing at least an urgingtowards at least one of a predetermined motion and a predeterminedposition of the body part dependent on the computer controlled seriallygenerated electrical signals.

The perception by the user may have a predetermined somatosensorysensation dependent on the computer controlled serially generatedelectrical signals. The haptic sensory cues may invoke the perception bystimulating a somatosensory system of a user comprising at least onereceptor including thermoreceptors, photoreceptors, mechanoreceptors andchemoreceptors to cause the user to perceive an experience of at leastone of proprioception (e.g., body part position and strength ofmovement), mechanoreception (e.g., touch), thermoception (e.g.,temperature), and nociception (e.g., pain).

In accordance with an aspect of the inventive HHMI, a method is providedfor using a human/machine interface. The method includes detecting theonset of an involuntary tremor of a user using a human/machineinterface. Electrical signals are determined having electricalcharacteristics effective to mitigate the involuntary tremor. Theelectrical signals are applied to the user using the human/machineinterface.

In accordance with another aspect of the invention, an apparatuscomprises at least one processor, and at least one memory includingcomputer program code. The at least one memory and the computer programcode are configured to, with the at least one processor, cause theapparatus at least to detect the onset of an involuntary tremor of auser using a human/machine interface. The electrical signals aredetermined having electrical characteristics effective to mitigate theinvoluntary tremor, and the electrical signals are applied to the userusing the human/machine interface.

In accordance with another aspect of the invention, a non-transitorycomputer readable memory medium stores computer program instructionswhich, when executed, perform operations for detecting the onset of aninvoluntary tremor of a user using a human/machine interface;determining electrical signals having electrical characteristicseffective to mitigate the involuntary tremor; and applying theelectrical signals to the user using the human/machine interface.

In accordance with another aspect of the invention, a plurality of firstsensory cues are generated capable of being perceived by a user. Theplurality of first sensory cue are time-sequentially generated andeffective for stimulating at least one sense of the user. A plurality ofhaptic sensory cues may be generated capable of being perceived by theuser. The haptic sensory cues may be received by the user dependent oncomputer controlled time-sequentially generated electrical signals. Theelectrical signals invoke a perception by the user related to the senseof touch. The haptic sensory cues may be generated in synchronizationdependent on the time-sequentially generated plurality of first sensorycues.

In accordance with another aspect of the invention, a plurality ofhaptic sensory cues are generated capable of being perceived by a user.The plurality of haptic sensory cues are dependent on a determinedcondition of at least one movable member of a performing body performingan event. The plurality of haptic sensory cues are effective forstimulating a touch processing center of a brain of the user based onthe determined condition. A plurality of visual sensory cues aregenerated capable of being displayed to the user on a video displaydevice. The visual sensory cues provide a virtual visual indication tothe user of a position of at least one of the at least one moveablemember and the performing body. The visual sensory cues are effectivefor stimulating the visual processing center of the brain of the user.The visual sensory cues may be synchronized with the haptic sensory cuesso that the position is virtually visually indicated in synchronizationwith the haptic sensory cues and so that the visual processing center isstimulated with the visual sensory cues in synchronization with thehaptic sensory cues stimulating the touch processing center.

In accordance with another aspect of the invention, electrical activityis received from at least one of muscles and nerves of a user. Anelectrical signal is determined having characteristics based on thereceived electrical activity. The electrical signal is generated andapplied to an object to cause an action dependent on the receivedelectrical activity. The object can be a biological component of theuser, another user, or a remotely located machine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an inventive Unmanned Vehicle System(UVS) interface;

FIG. 2 shows a block diagram of onboard condition sensing and control ofan inventive UVS;

FIG. 3 shows an individual conductive patch connected through atransistor to an x-y grid;

FIG. 4 shows a conductive patch applied to the skin surface of a userfor applying and detecting electrical signals to receptors, musclesand/or nerves of the user;

FIG. 5 shows a transducer connected to x-y conductors;

FIG. 6 shows the relatively smaller signal receiving transducers forsensing and relatively larger signal applying electrodes for applyingfeedback co-disposed in electrical communication with an x-y grid ofconductive leads;

FIG. 7 shows a plurality of transducers interconnected to an x-yconductive grid with one of the transducers being energized during asignal transmission scan;

FIG. 8 illustrates a user's bare arm;

FIG. 9 illustrates the arm without skin showing a location of electroderelative to the muscle groups of the arm;

FIG. 10 illustrates the arm with a sleeve of an inventive hapticinterface;

FIG. 11 illustrates the arm with gel electrodes targeting individualmuscles or muscle groups;

FIG. 12 illustrates the arm with the sleeve of the inventive hapticinterface including an x-y grid of relatively smaller signal receivingtransducers and relatively larger signal applying electrodes targetingindividual muscles or muscle groups;

FIG. 13 shows an arm of the user wearing the inventive haptic interfacetargeting specific muscle groups for applied electrical stimulation;

FIG. 14 shows the arm of the user wearing the inventive haptic interfacewith the targeted muscle groups involuntarily contracted;

FIG. 15 shows a UVS;

FIG. 16 shows a user wearing a system for applying audio/visual/hapticcues and for receiving control intention input via electrical signalsreceived from the user's body;

FIG. 17 shows a human/human interface where the haptic, visual and audioexperiences of one user is transferred to another user;

FIG. 18 shows an inventive human/machine interface composed of tactilefinger tips;

FIG. 19 shows an inventive human/machine interface comprising an orbhaving haptic and pressure active finger grooves;

FIG. 20 shows the inventive orb with high resolution haptic and pressureactive finger grooves;

FIG. 21 shows a block diagram of circuit components of the inventiveorb;

FIG. 22 shows elements of a hand applied wireless haptic informationtransducer of the inventive human/machine interface;

FIG. 23 illustrates an audio/visual/haptic signal collecting system;

FIG. 24 illustrates an audio/visual/haptic signal applying system;

FIG. 25(a) shows an artificial real-time perspective view of a UAV asdisplayed on a visual cue system of the inventive human/machineinterface;

FIG. 25(b) shows a 360 degree camera system for collecting videoinformation onboard a remote machine, such as a drone;

FIG. 26 illustrates a chair configured to receive and apply haptic andaudio signal;

FIG. 27 illustrates a visual sensory cue showing an actual tennis racketseen from the perspective of the user with an overlay of a virtualtennis ball generated using computer program code and displayed using anaugmented reality display, such as augmented reality eyeglasses;

FIG. 28(a) shows a UVS configured as a biomimicry bird at the start of apropulsion flap;

FIG. 28(b) shows the UVS configured as the biomimicry bird on a upwardstroke;

FIG. 28(c) shows the UVS configured as the biomimicry bird soaring;

FIG. 29(a) shows a bird with a control and communication circuit fixedto its back;

FIG. 29(b) shows a bird with the control and communication circuitblocking muscle signals from the brain of the bird and applying computercontrolled muscle signals to the flight muscles of the bird;

FIG. 29(c) illustrates the skeleton and feathers of a wing of a birdhaving sensors and transducers for remote computer-controlled flight;

FIG. 30(a) is a flow chart illustrating the steps for collecting datasets of a sequence of sensory activity of an event to be replicated,transmitted and/or recorded;

FIG. 30(b) is a flow chart illustrating the steps for generating datasets of a sequence of sensory activity of an event that has beencollected;

FIG. 31(a) is a perspective view showing an embodiment of a signalelectrode having conductive bumps; and

FIG. 31(b) is a cross section of mid-forearm showing conductive bumpsignal electrodes selectively applying and detecting electrical activityto muscles and nerves.

FIG. 32 is an illustration showing the muscles of the arm of a user;

FIG. 33 is an illustration showing an exemplary embodiment of theinventive haptic human/machine interface configured as a sleeve disposedon the arm of the user;

FIG. 34 schematically illustrates an electrode equivalent electroniccircuit for applying and detecting electrical signals;

FIG. 35 illustrates pulse square waves depicting computer generatedelectrical signals that can be selectively applied via the electrodeequivalent electronic circuit;

FIG. 36 shows the electrode equivalent electronic circuit applying aselected electrical signal applied to selected electrodes;

FIG. 37 illustrates an electronic circuit including a plurality ofaddressable electrodes for applying and/or detecting electrical signalsto muscle fibers and nerves;

FIG. 38 illustrates another electronic circuit including a plurality ofaddressable electrodes for applying and/or detecting electrical signalsto muscle fibers and nerves;

FIG. 39 is a schematic of another electronic circuit example having aPWM driver for selectively applying a pulse width modulated AC or DChaptic electrical signal to selected addressable electrodes;

FIG. 40 is a schematic showing the electronic circuit example forapplying the electrical signal through muscle and nerve fibers through aplurality of individually addressable electrodes;

FIG. 41 is a schematic showing a repeatable circuit element forindividually addressing a respective electrode and a correspondingelectrode of a plurality of electrodes to selectively apply, detect orswitch off signals to the addressable electrodes;

FIG. 42 illustrate an exemplary embodiment including an electroniccircuit for detecting electrical activity of muscles and nerves from aplurality of electrodes;

FIG. 43 shows the muscles and bones of the forearm and hand of a user;

FIG. 44 shows the forearm and hand of the user illustrating an exemplaryembodiment of the inventive HHMI;

FIG. 45 illustrates detected electrical signals indicating the onset ofa tremor and the application of electrical signals applied to tremormitigation muscles;

FIG. 46 shows the locations of the muscles having the detected andapplied electrical signals;

FIG. 47 is a close up cross section showing an embodiment of anelectrode for use with the inventive HHMI having conductive loops;

FIG. 48 is a close up cross section showing an embodiment of anelectrode for use with the inventive HHMI having conductive stems;

FIG. 49 is a cross section showing the embodiment of an electrode foruse with the inventive HHMI having conductive loops;

FIG. 50 is a cross section showing the embodiment of an electrode foruse with the inventive HHMI having conductive stems;

FIG. 51 is a perspective view showing an individually addressableelectrode for use with the inventive HHMI having conductive stems;

FIG. 52 is a perspective view showing conductive stems of theindividually addressable electrode for use with the inventive HHMI;

FIG. 53 is a perspective view showing an individually addressableelectrode for use with the inventive HHMI having conductive hemispheres;

FIG. 54 is a cross section showing conductive hemispheres of theindividually addressable electrode for use with the inventive HHMI;

FIG. 55 illustrates an embodiment of the inventive HHMI configured as asleeve having addressable electrodes connected via a grid of x and yelectrodes;

FIG. 56 is a flow chart showing a calibration algorithm for calibratingthe HHMI to an individual user's body;

FIG. 57 is a flow chart showing a refinement algorithm for refining thecalibration of the HHMI;

FIG. 58 shows the muscles of a hand of the user;

FIG. 59 shows the inventive HHMI configured as a pair of gloves;

FIG. 60 shows the mapping of individually addressable electrodes themuscles of the hand of the user;

FIG. 61 shows the inventive HHMI configured as an undergarment andhaving clusters of more denser packed electrodes and clusters of lessdenser packed electrodes;

FIG. 62 illustrates a use of the inventive HHMI as a component of anaccelerated learning system for teaching a musical instrument;

FIG. 63 illustrates the basic HHMI signal detection and applicationcomponents;

FIG. 64 shows data collection on a bicycle for use in sports training;

FIG. 65 shows the synchronized application of sensory cues dependent onthe data collection of FIG. 64 during a training session;

FIG. 66 shows the collection of data sampled along a route taken by acyclist;

FIG. 67 is an isolated view of the collection of data sampled along theroute showing the bicycle at an angle and height relative to sea level;

FIG. 68 is graph showing the collection of data as exemplary altitudeand angle relative to sea level data collected over time along the routetaken by the cyclist;

FIG. 69 illustrates a chair configured for an exemplary entertainmentapplication;

FIG. 70 illustrates an augmented reality visual sensory cue showing anactual tennis racket seen from the perspective of the user with an videooverlay of a virtual tennis ball generated using computer program codeand displayed using an augmented reality display, such as augmentedreality goggles;

FIG. 71 shows a user experiencing deep immersion of a virtual reality, ablock diagram showing detection and application of data, andillustrating the processing centers of the brain stimulated by theapplied synchronized sensory cues;

FIG. 72 shows the inventive HHMI configured as a glove having higherdensity, higher resolution, smaller electrodes disposed at nerve-richfinger tips of the user;

FIG. 73 shows the inventive HHMI configured as a sleeve and applied as aretrofit modification or OEM device in signal communication with agaming controller;

FIG. 74 illustrates a virtual reality controller having haptic pads forapplying electro-tactile sensations to the finger tips of a user;

FIG. 75 is a flow chart illustrating an algorithm for detecting dataincluding user-applied pressure, bio-generated electrical signals,bio-active electrical signals, and changes in position and accelerationsof the virtual reality controller;

FIG. 76 illustrates the inventive HHMI with synchronized haptic, audioand video signals dependent on a virtual or augmented reality anddependent on actions of a remotely located second user for creating ahuman/human interface;

FIG. 77 illustrates the inventive HHMI for remote sensing andcontrolling of a drone;

FIG. 78 illustrates the inventive HHMI configured as a full body suitmapped to a remote drone, and including haptic, audio and video sensorycue systems, body position and electrical activity sensing systems andbrain activity sensing system;

FIG. 79 illustrates the inventive HHMI configured for applyingelectrical stimulation to large muscle groups to provide haptic cues ofa maimed or unmanned aerial vehicle;

FIG. 80 shows the plurality of drones having proximity sensors fordetecting other drones, ground, and other potential collision obstacles;

FIG. 81 is a flow chart showing an algorithm for data collection;

FIG. 82 is a configuration of the inventive HHMI configured for roboticsurgery;

FIG. 83 shows a sport apparatus configured as a sensory data sensingshoulder pads worn by a football player;

FIG. 84 shows a sport apparatus configured as a sensory data sensinghelmet worn by a football player;

FIG. 85 illustrates sensory data detected by an on-field player, wherethe data is applied as mapped sensory cues for virtual immersion by afootball fan;

FIG. 86 shows sports apparatus configured as a sensory data sensinghelmet, glove, baseball and bat used by a baseball player;

FIG. 87 shows the HHMI can be configured with a gyroscope showingelectrodes placed on Balance Control Muscles (BCM); and

FIG. 88 shows the HHMI can be configured with a Gyro-Vest that holds acore-stabilizing gyroscope adjacent to the chest of the wearer. thatholds a core-stabilizing gyroscope adjacent to the chest of the wearer.

DETAILED DESCRIPTION OF THE INVENTION

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any embodiment described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments. All of the embodiments described inthis Detailed Description are exemplary embodiments provided to enablepersons skilled in the art to make or use the invention and not to limitthe scope of the invention which is defined by the claims.

The elements, construction, apparatus, methods, programs and algorithmsdescribed with reference to the various exemplary embodiments and usesdescribed herein may be employable as appropriate to other uses andembodiments of the HHMI, some of which are also described herein, otherswill be apparent when the described and inherent features of the HHMIare considered.

In accordance with an embodiment of the inventive accelerated learningsystem, a multi-sensory, virtual reality, drone interface isolates apilot from ambient distractions and immerses the pilot incomputer-controlled and synchronized auditory, visual and hapticstimulation enabling intuitive remote controlled flight as if the pilothas become the drone. Haptic cues applied as electrical signals on theskin invoke involuntary muscle contractions perceived as though causedby an external mechanical force. Synchronizing haptic cues withreal-time audio/visual cues fully immerses the pilot within thein-flight drone environment.

Many of the embodiments described herein are exemplified by a pilot(human) controlling a drone (machine). However, the inventivehuman/machine or human/human (either referred to herein as the other)interface is adaptable for use in other interactions between anindividual and another body. The inventive interface is applicable to awide range of techniques and applications, including, but not limited toentertainment, sporting, military, gaming, computer control, homeautomation, space and deep sea probes, as well as the remote controldrone or robot operation. The inventive interface can also provide animmersive way to communicate between two people remotely located fromeach other, or to experience an activity being performed or observed byanother, in real time and from previously detected and recorded data.

The haptic sensory cue can be a vibration applied to the predeterminedfinger. The vibration can have an intensity corresponding to an audiovolume and/or pitch, and/or to a visual color and/or intensity. Thehaptic sensory cue can be an electrical impulse applied to a nerve ormuscle corresponding to predetermined muscles and nerves being targetedfor response. The visual indication can be displayed on a displaycomprising at least one of a pair of reality augmentation/virtualreality eyeglasses or goggles, a computer monitor, a television, a smartphone display or a personal information device display. The visualindication may comprise at least one of a color image, light intensityor light position displayed on a display. The visual information maycomprise any image viewed by the user.

FIG. 1 shows a block diagram of an inventive human/machine interface.FIG. 2 shows a block diagram of onboard condition sensing and control ofan inventive UVS. In accordance with an exemplary, non-limitingembodiment, an Unmanned Aerial System or Unmanned Robotic System (UVS)pilot interface includes computer-controlled sensory stimulation used toconvey and enhance remotely sensed ambient and vehicle stresses duringpilot-controlled drone flight. To go beyond conventionaljoystick/monitor control, a much more fully immersive experience iscreated through a unique multi-sensory human/machine interface.

FIG. 3 shows an individual conductive patch connected through atransistor to an x-y grid. The inventive human/machine interfaceprovides neuromuscular electrical stimulation (NMES) to motor andsensory nerves and muscles to produce muscular contractions that createhaptic sensory cues. These “touch” cues are synchronized with one ormore other sensory cue(s) (e g, immersive 3D visual and audio). If theconductive paths provide sufficient S/N of the electrical activity fromthe user, the circuit can be simplified and avoid having to amplify inproximity with each conductive patch. The x and y sensor pads can detectelectrical changes in the nerve fibers and muscles fibers to determinethe location of an individual user's specific areas of maximum actuationfor a given movement. The same x and y grid can be used to applyelectrical pulses to stimulate the nerves and/or muscle fibers, or tosimulate sensation felt at receptors in the skin.

The inventive human/machine interface is a light weight, wireless, highresolution electrical signal sensing/applying mechanisms for thedetection of the pilot's control intentions (to control the droneflight) and for the application of enhanced haptic cues (to experiencethe drone's flight conditions).

FIG. 4 shows a conductive patch applied to the skin surface of a userfor applying and detecting electrical signals to receptors, musclesand/or nerves of the user. FIG. 5 shows a transducer connected to x-yconductors. FIG. 6 shows the relatively smaller signal receivingtransducers for sensing and relatively larger signal applying electrodesfor applying feedback co-disposed in electrical communication with anx-y grid of conductive leads. FIG. 7 shows a plurality of transducersinterconnected to an x-y conductive grid with one of the transducersbeing energized during a signal transmission scan.

A flexible grid of x-y conductors enable multiplexed, high resolutionsignals to be detected and applied. For example, the electrical activityof the body of the user (particularly, the nerves and muscle) can bedetected and used to determine the user's control intentions. Thesecontrol intentions can be machine-implemented actions such as moving acursor on a display screen, selecting a button for a hyperlink in anHTML document, controlling home automation equipment, gaming, remotecontrol of unmanned vehicles, control of deep space and deep sea probes,etc. The grid of x-y conductors form intersections. At the x-yintersections, a haptic transducer can receive and apply electricalsignals. For example, as shown in FIG. 3, a conductive patch can be usedto apply and to receive electrical signals to and from the body of theuser. Schematically represented are transistors associated with theconductive path that control the flow of electrons (acting as a switch)and that amplify the flow of electrons (acting as an amplifier). Theelectric circuit shown is simplified, and in actual practice additionalcircuit elements, such as capacitors, resistors, other transistors andother electronic elements may be included. Each conductive patch can beindividually addressable enabling, for example, high resolution signaldetecting and application using electrode scan techniques similar tothose used for an active matrix or non-active matrix video display.Extremely high resolution can be achieved to precisely map the sourcesof electrical activity (the subcutaneous muscles and nerves). Thisdetected electrical activity provides a determination of the bestlocations to detect and to apply the electrical signals for a particularuser. The detected electrical activity can also be used to convey theuser's control intentions in the human/machine interface. Transistorscan be used to provide selective conductive pathways through the nervefibers and muscles using, for example, techniques, components andsystems similar to a scanning x-y display, such as a OLED or LEDdisplay, or a capacitive touch sensitive display.

The inventive haptic human/machine interface uses involuntary motorcontrol (pilot) from a locally generated signal (interface) that isdependent on a remotely transmitted signal (drone) to create animmersive virtual reality experience. In a non-limiting exemplaryembodiment, the conductive patches apply a phased DC voltage to theskin. For example, a microprocessor may vary which patches (e.g., usinga conductive x-y or multiplexed grid) apply the haptic signal to thepilot's arm. The conductive patches can be placed and shaped to targetthe muscles/nerves that control finger/hand/forearm movement withaccurate and repeatable controlled movements. The involuntary movementcan be applied to feel like a varying resistance (like wind pressure) ora more active force (like a bump). Even a subtle action on the remotedrone can be sensed, where the pilot feels the artificially createdsensation of an applied resistance, like flying one's hand through theair out the car window. Sufficiently high resolution of the detectedelectrical activity can be achieved to precisely map the sources ofelectrical activity (and hence determine the best locations to apply theelectrical stimulations and obtain precise control signals for aparticular pilot).

The same conductive patches can detect and apply electrical activity, sothe user's body (e.g., forearm and hand) can be mapped to calibrate thepatches and signals for a particular pilot. The pilot's movements aretranslated to drone flight control signals. For example, the angle of adrone control surface can be changed in response to the pilot pushingback on the apparent resistance.

A chair (shown in FIG. 26) can be part of the human/machine interface,providing a comfortable entertainment, teaching or remote controlexperience with immersive 3D/360 video, binaural audio andhaptic-feedback. In the VR scene, the chair could appear transparent tomaximize the pilot's view.

In a medical use example the haptic feedback can be in direct responseto the user's movements. This is useful, for example, to counteractinvoluntary tremors caused by Parkinson's disease. In this case, theconductive patches (and/or mechanical sensors such as strain gauges) canbe used to detect electrical and/or muscle activity of a tremor. Thedetected activity is analyzed by a microprocessor and a counteractingelectrical signal is applied. The counteracting electrical signal can beapplied to disrupt the detected activity by causing a counteractingmuscle response or confusing the tremor muscles. Another medical use canbe a tactile suit used to treat autism and other conditions where anapplied tactile sensation can provide benefit and relief. It is knownthat some affected by autism benefit from a controlled tactilesensation, such as light pressure squeezing on parts of the body. It istypical to apply this pressure using a pressure or weight vest. Theinventive tactile suit can be configured to provide the sensationreplicating light pressure or other stimulation and thereby provide thebenefit, for example, of light pressure in a custom-calibrated, mobileand convenient system.

For example, electrical activity is received from at least one ofmuscles and nerves of a user, for example, using the haptic interfacecomponents shown and described herein. A control signal is determined,for example, using a microprocessor, having characteristics based on thereceived electrical activity. The control signal is generated, forexample, by an electronic circuit capable of generating a TENS or NMESelectrical signal. The control signal is applied to an object, such asthe user's arm, to cause an action, such as an involuntary musclemovement, dependent on the received electrical activity. As anon-limiting example, the received electrical activity may be the resultof an involuntary tremor of a user having Parkinson's disease. Thecharacteristics of the control signal are determined based on theinvoluntary tremor to cause involuntary muscle movement that counteractsthe involuntary tremor. The control signal is generated as an electricalsignal having the characteristic to cause the involuntary musclemovement that counteracts the involuntary tremor, and the control signalis applied to the user to cause the muscle movements that counteract theinvoluntary tremor.

FIG. 3 shows a simplified circuit and block diagram of an individualconductive patch connected through a transistor to an x-y grid forreceiving electrical activity to apply to the user, and for receivingand amplifying electrical activity from the user. For example, theelectrical activity can be applied to stimulate somatic and kinestheticsensations related to force and touch. Somatic sensation, for example,are perceived cutaneous (skin) and subcutaneous (below skin).Kinesthetic sensation are more related to mechanical body parts, such asjoints and muscles. In general, these sensations can be called hapticfeedback which is naturally used to determine things like geometry,roughness, slipperiness and temperature, weight and inertia (force). Ina non-limiting exemplary utilization of the embodiments describedherein, in an “accelerated learning mode”, the sensory stimulation isapplied as haptic (touch), visual and audio cues applied to the sensesof a student, where the sensory cues correspond to a performance beinglearned (e.g., piloting a drone). The sensory cues replicate and/oraugment the tactile, visual and audio sensations experienced during thecontrol of an actual drone flight. An enchanted flight simulator isobtained where the student pilot experiences the visual and audioinformation associated with the control of the drone, with the additionof haptic sensations that create the muscle-memory necessary for alearned action to quickly become an instinctive response. In the case ofa “performance mode”, such as an actual remote drone flight, the sensorycues provide real-time feedback of the ambient environment and stresseson the aircraft.

A non-limiting exemplary embodiment of an inventive haptic interface isconfigured as a sleeve that can be worn by a user, with the detectionand application of electrical signal activity obtained through auser-calibrated grid of conductive patches or electrodes. FIG. 8illustrates a user's bare arm. FIG. 9 illustrates the arm without skinshowing a location of electrode relative to the muscle groups of thearm. FIG. 10 illustrates the arm with a sleeve of an inventive hapticinterface. FIG. 11 illustrates the arm with gel electrodes targetingindividual muscles or muscle groups.

FIG. 12 illustrates the arm with the sleeve of the inventive hapticinterface including an x-y grid of relatively smaller signal receivingtransducers and relatively larger signal applying electrodes targetingindividual muscles or muscle groups. FIG. 13 shows an arm of the userwearing the inventive haptic interface targeting specific muscle groupsfor applied electrical stimulation. FIG. 14 shows the arm of the userwearing the inventive haptic interface with the targeted muscle groupsinvoluntarily contracted.

The haptic interface may be in the form of a comfortable, easily worngarment that the pilot wears with little or no restriction of movement.Although a full body garment could create more full tactile immersions,the pilot interface requires direct contact with only the arm of thepilot to be effective.

A sleeve of the garment, as shown, for example, in FIGS. 10, 12 and13-14 is constructed having a flexible grid of x-y conductors, where atthe x-y intersections a haptic transducer can receive and applyelectrical signals. Extremely high resolution can be achieved toprecisely map the sources of electrical activity (the subcutaneousmuscles and nerves) and hence determine the best locations to detect andto apply the electrical signals for a particular pilot.

Since every human body is different, in a calibration mode the pilotperforms a known task that causes nerve firings and muscle contractions,such as a motion that replicates using the hand as a control surface,e.g., a flap or thruster. In this case, the known task can be a handmotion forming a flat plane with the fingers and bending at the wrist asif deflecting air. The characteristics of the body-generated electricalactivity (e.g., electromyographic signals generated by the nerves andmuscles as the hand is formed into a plane and bent at the wrist) aresensed by the x-y transducer grid and used to calibrate the location,relative strength, etc. of each detected electrical signal. In additionto the body-generated electrical activity, other physiological changescan be detected, such as a change in the shape of the user's arm causedby muscle contractions. These physiological changes are useful forcalibrating the inventive human/machine interface and also fordetermining pilot's intended control signals. The electrical and muscleactivity that is detected and used for calibration, control intentions,user conditions, etc., can include EKG, EMG and EEG, as non-limitingexamples.

In an auto-action mode, the calibration data is used to determine thecharacteristics of the computer-generated electrical activity causing adesired automatic and involuntary movement of the pilot's body parts.The result is the pilot perceives the involuntary movement as thoughcaused by an externally applied force, in this case, as through thepilot's hand is the in-flight control surface deflecting air.

Electrical stimulation is applied through the skin on at least one ofthe arms of the pilot dependent on a desired position to be achieved bythe pilot's hand and arm. The desired body position can be related to asensed parameter, such as flex, rotation, tilt, pitch, yaw, temperature,vibration, and other detectable stresses or conditions of a mechanicalcomponent (wing, fuselage, control surfaces, etc.) of the UVS. Thesensed parameter could be air pressure experienced at a wing controlsurface while maneuvering. The sensed parameter is transmitted from thedrone (using RF or line-of-sight optical), causing a computer-controlledNMES cue (electrical stimulation) resulting in an auto-action responsein the hand of the pilot feeling pressure to assume a position directlyrelated to the drone's control surface. The pressure to move the hand isthe result of muscle movements caused by the NMES cue. The pilotexperiences the sensation of resistance or pressure because of thecomputer controlled electrical signals applied to the pilot's ownsensory/muscular physiology. In addition to pressure and resistance, thephysical sensation of vibrations, knocks and even scratches can beperceived as the result of subcutaneous controlled electrical signalstimulation. The muscle movements are involuntarily and automatic. Thereare no mechanical force simulators involved, although there can be.Vibration, for example, can be stimulated by both the applied electricalsignal and mechanical buzzers (or rumble packs, etc.) that can beapplied, for example, from a “massage” chair or from a transducerassociated with one or more of the x-y interfaces. In the case of musicand entertainment, for example, the transducer could deliver thevibration as low end bass notes, while applied electrical signaldelivers the sensation of light scratches corresponding to higher notes.Bass beats, for example, could be perceived through a knock sensationresulting from an appropriately controlled electrical signal.

The hands of a human are particularly sensitive to haptic stimulation.For example, the muscles that move the finger joints are in the palm andforearm. Muscles of the fingers can be subdivided into extrinsic andintrinsic muscles. The extrinsic muscles are the long flexors andextensors. They are called extrinsic because the muscle belly is locatedon the forearm. The application of haptic sensation, such as the hapticsensory cues, can be applied to various parts of the body, and theinventive accelerated learning system adapted to enable a wide range ofapplications, from remote control operation to human/machine interfaceto teaching to entertainment to rehabilitation. By noting thesensitivity to stimulation of the body parts (e.g., the fingertips arevery perceptive to tactile stimulation), the application of hapticsensory cues can be selective in accordance with a desired interface,learning or entertainment enhancement. For example, the fingers (and/orthe muscles controlling the fingers and/or the nerves communication withthose muscles) can receive haptic stimulation in the form of a pressure,vibration, electrical impulse or other stimulation.

The inventive human/machine interface includes a virtual reality visualsystem that deepens the immersion for the pilot by tying in real-timehead and body movements to a three dimensional, perceived visual sphere.Camera systems onboard the drone feed real-time video from enough cameraangles to create a seamless (after software cleanup) sphere of vision.As an example, if the pilot is sitting in the inventive chair shown inFIG. 26, this virtual visual sphere could give the pilot the impressionthat he is flying a glass chair rather than a drone.

The audio system of the inventive UVS interface may include highquality, binaural, audio provided through sound canceling headphones toreplicate the actual, real-time sounds that are ambient to the remoteUVS, or other sounds such as white noise, soothing or aggressive music,“spotter” updates and command instructions. The inventive human/machineinterface is intended to isolate the pilot from local ambientdistractions, but the degree of this isolation can be easily controlledto maintain safe conditions. Also, although a bit more invasive thansurface electrodes, the electrodes used to apply or detect theelectrical signals can be of a type where the skin is pierced. However,piercing the skin is not necessary to effect benefits from the inventivehuman/machine interface and much less invasive gels, gel electrodes,carbon fiber electrodes, etc., can be used.

In addition to the full immersion of visual and auditory stimulationcorresponding to the remote drone as it flies, the application ofauto-action and other haptic cues enable the pilot, in a sense, tointimately “feel” the flight conditions experienced by the remote UVS.With the level of immersion into the real-time conditions of the UVScreated by the inventive human/machine interface, the pilot does notjust feel like he or she is flying the UVS, to the extent possible, thepilot “becomes” the UVS.

For example, the haptic cues cause the pilot to experience wind gusts assudden jarring movements, or unbalanced stresses on the mechanical andcontrol surfaces, such as experienced in a tight banking maneuver, asproportionally applied pressure or resistance to movement. Even subtlenuances such as the warmth of the sun shining on the top surfaces of thedrone can be experienced at a corresponding location on the back of thepilot. It isn't yet known what degree of immersion and which nuancesmight be optimal for a given set of circumstances, the inventivehuman/machine interface is designed with the intention of enabling thehigh quality resolution of multiple computer generated, enhanced andreal-time synchronously applied, immersive sensory cues.

Thus, forces experienced, for example, by the drone, are detected andtransmitted, then converted to proportional electrical signals. Thepilot's body's sensation receptors such as, nocireceptorsmechanoreceptors, and thermoreceptors including proprioceptors andchemical receptors, receive the computer controlled haptic cues appliedas electrical stimulation to replicate, for example, natural sensationsreceived by the human body through the skin, muscles and bones. Sincethe nervous system of the human body operates via electrical impulses,any nerve, nerve ending, muscle or receptor can be triggered byelectrical stimulation. Signal characteristics such as; the location,timing, pulse length, frequency and amplitude of the electricalstimulation are applied under the control of the computer depending onthe intended type of sensation or muscle movement to indicate to thepilot the drone's onboard and ambient conditions.

Depending on the applied NMES cue, the pilot experiences the haptic cueas pressure, as if pushing against resistance and/or being forced tomove into the position related to the wing control surface, and/or avibration or even a blow as if being jarred by an external force (e.g.,being buffeted from a wind gust).

The inventive human/machine interface has an advanced multi-sensorysystem that uses the physiology of the pilot to integrate the onboardand ambient conditions of a remotely flown drone into the informationpool used by the pilot to control the drone's flight.

FIG. 13 shows an example where a specific muscle (bicep) is targeted forcontraction by applying a TENS type transcutaneous electrical signal.The electrical signal is applied as a DC voltage between a firstelectrode group and a second electrode group. The appropriate electrodegroup to invoke a desired muscle response can be determined for example,during a calibration mode. During the calibration mode, these same firstelectrode group and second electrode group are used to detect theelectrical activity generated when the user performs a known action,such as raising the hand to the chest (contracting the bicep muscle).Additionally or alternatively, the appropriate electrode groups toinvoke a desired muscle response can be extrapolated from thecalibration data because the general physiology of a human arm is wellknow. In this case, the calibration mode provides fine tuning of apredetermined electrode pattern, where the predetermined electrodepattern is based on human physiology and the fine tuning is based on theparticular electrical activity detected for the user during thecalibration mode. A strain gauge wire can be used to detect musclemovement and/or a memory metal used to contract and apply a squeezingforce, acting as conductive pathways of the x/y grid or provided asseparate components.

The inventive haptic interface using sensory feedback and algorithms tocorrectly control the characteristics of the electrical stimulation,muscle contractions can be induced that result in same movements of thebody part of the user (e.g., fingers) as if performed voluntarily by theuser using precisely controlled muscle movements.

Muscle contractions and changes in body part positions can be used asmetrics during calibration and also to obtain feedback while the appliedelectrical stimulation causes an automatic and involuntary movement ofthe user's body parts. The sleeve may include transducers can be used tomeasure changes in muscle shape or body part position, or to apply apressure, such as a squeeze or vibration. For example, a shape memoryalloy (which could be formed as a sheath around or otherwise incommunication with the x-y conductors) can be used under control ofelectrical signals from the computer, to apply haptic cues in the formof pressure or vibration.

Neuromuscular electrical stimulation is applied as a low frequency,relatively high intensity pulse. The pulse, which may be biphasic,triggers the alpha motor nerves that cause muscle movement. The higherthe intensity of the electrical stimulus, the more muscle fibers will beexcited, resulting in a stronger contraction. The contraction can havedifferent speeds and duration of the contraction dependent on thecharacteristics of the applied electrical signal. The characteristics ofthe applied electrical signal can be controlled to cause isometricand/or isotonic muscle contraction, where an isometric musclecontraction leads to a tension in a muscle, without changing the lengthof the muscle, and isotonic muscle contraction, results in a shorteningof the muscle. In accordance with the inventive haptic interface, acomputer controls the characteristics of electrical signals applied to,for example, the motor neurons of the user's nervous system to cause adesired sensation and/or muscle movement. Exciting the motor neurons viathe body's nervous system produces exactly the same result as when theneurons are excited through the computer controlled electricalstimulation.

In accordance with the inventive haptic interface, computer controlledelectrical signals can be applied with signal characteristics effectiveto stimulate one or more of the tactile receptors found in the skin. Thesignal characteristics are controlled to selectively stimulate thereceptors that have, for example, different receptive fields (1-1000mm2) and frequency ranges (0.4-800 Hz). For example, broadreceptive-field receptors like the Pacinian corpuscle produce vibrationtickle sensations. Small field receptors such as the Merkel's cells,produce pressure sensations.

In a teaching scenario, in a general embodiment, the NMES is applied asthe generated sensory cue to the user dependent on the position of abody part of a performer relative to a performance element of aperformance object with which an event is performed. In a more specificembodiment, such as simulated flight training of maimed or unmannedaerial vehicles, one or more sensory cues are computer controlled tostimulate the sense organs of the user (e.g., student pilot) effectivefor stimulating various processing center of a brain of the user so thatuser learns how to position his body member corresponding to theposition of the performer of the event. Sensory cues are applied to theuser and are dependent on a position of at least one body member of aperformer relative to a performance element of a performance object withwhich an event is performed. For example, in addition to the haptic cue,audio and visual sensory cues can be applied synchronously to the user'ssenses. The sensory cues are effective for stimulating the variousprocessing center of a brain of the user so that user learns how to, forexample, rapidly achieve the position of a particular body member (e.g.,hand on a joystick) corresponding to the position of an instructor orperformer (e.g., actual pilot) performing the event (e.g., flying anactual plane or drone).

FIG. 15 shows a UVS. The inventive accelerated learning system can beused to teach and/or improve hand-eye coordination for a variety ofactivities, including, but not limited to, video and online gaming, aswell as remote control of devices, such as military drones and the like.

In the case of military drones, it is desirable that the operators begiven much time at the controls of the remote drone in order to learnthe subtleties of remote controlling a drone or robot. For example, inthe case of a flying drone, the operators can be provided with a flightsimulation so that the cost and time involved in flying an actual droneis avoided. The operator can also be given a more immersive experiencewithout having to fly the actual drone. In this case, the operator mayuse a recorded actual drone mission, and receive haptic, visual andaudio cues that replicate the experience of the remote drone operatorduring the actual mission. The actual mission can include apredetermined course, so that the operator knows what to anticipatebefore the haptic audio and visual cues are applied. For example, theset course may include a series of banking and turning maneuvers and/ortake off and landing.

The inventive accelerated learning system may be particularly useful formilitary instruction. For example, as military technology progresses,there is an increasing emphasis on the use of remote control devices,such as robots and drones to replace operators and soldiers and othermilitary personnel in the field.

Robot and drone use is becoming increasingly advantageous for otherapplications, such as law enforcement. Further, it is likely thatcivilian entertainment and other uses will become more and moredependent on the remote control of devices. Also, remote explorationsuch as deep-sea and space exploration will increasingly rely heavily onremote sensing/control of robotic systems.

The drones can be equipped with sensors, so that real time telepathy ofmotions and other sensory cues such as vibrations caused by wind gustsor banking of the drones wings, can be translated into haptic sensorycues applied to the remote drone operator.

The sensory cues translated from sensors on board the drone can also beapplied as audio and/or visual cues. Thus, the remote drone operator isable to perceive different aspects of the drone flight performancethrough various sensors and sensory cues. Because the different sensorycues are stimulating different parts of the operator's brain, theoperator is able to process the information in a manner which may bemore optimal then if the operator were to simply feel, for example, arumble-pack type vibration simulating the buffeting of the drone causedby wind currents. That is, the onboard vibration, or banking,acceleration, etc., experienced by the drone can be sensed using onboardsensors, and the telepathy of those sensors received and used to providesensory stimulation to the remote drone operator. The sensorystimulation may be, as just one example, audio and visual cues appliedto the operator to stimulate various parts of the operator's brain as anindication of the drone's performance. Through consistent combinedsensory stimulation, the operator receives enhanced learning of thesubtleties of the drone's performance in relation to external factors,such as wind, altitude and air temperature, and the operator's control.For example, if the operator's control would result in a stall, anonboard tilt sensor can provide telepathy indicating that the wing ofthe drone has an angle of attack that will result in an imminent stall.This telepathy can be converted into an audio and visual warning toindicate to the operator that a corrective action should be taken toprevent the stall.

More than just receiving an audio and visual warning, in accordance withthe inventive accelerated learning system, these sensory cues can bereceived in addition to haptic cues and electrical impulses applied toone or more area of the operator's body, to create a strong learnedbehavior/skills reinforcement in a highly immersive and convenientmanner. The remote control of a flying drone is an example of a use ofan embodiment of the inventive accelerated learning system. A pluralityof first sensory cues are generated capable of being perceived by auser. Each first sensory cue of the plurality of first sensory cues isdependent on a position of at least one body member of a performerrelative to a performance element of a performance object with which anevent is performed. In this case, the performer can be an actual pilotof a drone aircraft and the actual pilot's responses and control of theremotely controllable drone can be recorded to provide the sensory cuesto the user (e.g., a student pilot). Alternatively, artificialintelligence can be used to determine how a virtual pilot would react,for example, in a combat, take off, landing, or poor weather situation,and in this case the performer is a computer generated virtualperformer. Whether they are dependent on an actual performer or avirtual performer, when perceived by the user, the plurality of firstsensory cues are effective for stimulating a first processing center ofa brain of the user. For example, in the case of the flying of a droneaircraft, the position of the hands, fingers and/or feet of the actualor virtual pilot can be determined relative to a joystick, buttonsand/or other controllers of the remote controller used to perform theevent of actually or virtually flying the drone.

A plurality of visual sensory cues capable of being displayed to theuser on a video display device are also generated. For example, thevisual sensory cues can be dependent on signals from a video camera onan actual drone, or dependent on computer generated video images. Thevisual sensory cues provide a virtual visual indication to the user ofthe position of the at least one body member. For example, the virtualvisual indication can be the reaction of the drone to the body memberposition, and/or they can be the position of the actual or virtualperformers body member relative to the controls. As described elsewhereherein, two or more images can be displayed simultaneously to the usereither as an overlay (one image over the other) or side by side. Thevisual sensory cues are effective for stimulating the visual processingcenter of the brain of the user. The visual sensory cues aresynchronized with the first sensory cues so that the position of the atleast one body member is virtually visually indicated in synchronizationwith the first sensory cue and so that the visual processing center isstimulated with a visual sensory cue in synchronization with a firstsensory cue stimulating the first processing center. The synchronizedstimulation of the first processing center and the visual processingcenter is effective for teaching the user to perform a version of theevent. That is, the user receives the sensory cues related to the actualor virtual performed event, and these sensory cues are effective tocreate memory associations in the brain of the user so that the userlearns how to perform a version of the event.

A second plurality of sensory cues capable of being perceived by theuser can also be generated. Each second sensory cue of the plurality ofsensory cues is dependent on at least one of the position of the atleast one body member and an action of the event. The action dependenton the position of the at least one body member. In other words, as anexample, the action in this case can be how the remotely controlleddrone reacts to the position of the hand gripping the joystick thatcontrols the drone. The second sensory cues are effective forstimulating at least a second processing center of the brain of theuser. The second sensory cues are synchronized with the first sensorycues so that the second processing center is stimulated with a secondsensory cue in synchronization with a first sensory cue stimulating thefirst processing center. The synchronized stimulation of the firstprocessing center, the visual processing center and the secondprocessing center is effective for teaching the user to perform aversion of the event. For example, haptic or electrical stimulation canbe used as the second plurality of sensory cues. In this case, themuscles and/or nerves that control the muscles are stimulatedcorresponding to the position of the body member(s) or the actual orvirtual drone pilot. As an example, if during a real combat mission anactual pilot of a drone is forced to deploy a weapon in reaction to avisual indication provided from the drone camera, and/or an audiblecommand indicating hostile forces are acting against friendly troops thedrone is protecting, the actual pilots reaction to the visual indicationand/or command can be provided along with the same visualindication/command to the student pilot so that the student pilot learnsduring a training exercise of the correct response against the hostileforces needed to protect the troops.

The video display device can comprise at least one of a pair ofaugmented and/or virtual eyeglasses, a computer monitor, a television, asmart phone display or a personal information device display. Forexample, in the case of the eyeglasses, a device such as google glasscan be used to record the body member position of the actual pilotduring the actual drone flight, providing that pilots perspective andindicating when he looks down at his hands, up at a display screen orinstrument, and even what portion of the screen or instrument or whatscreen or instrument is viewed during the reaction to a particularflight situation. The user during the learning session is then given thesame visual information in the form of the virtual visual cues. Themuscles and/or nerves that control the movement of the head and even themuscles controlling the movement and focus of the eyes can be stimulatedin synchronization to the visual cues so that the muscle memory iscreated in the association among the different brain processing centers.

As described herein, and as will be logically foreseeable to oneordinarily skilled in the art from the teachings herein, the event canbe many different activities and actions, including controlling at leastone of a sports related object, a musical instrument, a weapon, a videogaming controller, a remotely controllable system including a spaceprobe, a drone aircraft, an underwater probe, a robot. Also, at leastone of the first and the second plurality of sensory cues are remotelydetermined from corresponding to the event that is performed, the eventbeing remote in at least one of time and location relative to the user;and wherein at least one of the first and the second plurality ofsensory cues stimulates a brain processing center for at least one ofthe five senses of hearing, seeing, smelling, feeling and taste.

The flight controls, for example, controlling a drone can be enhancedbeyond the conventional joystick operation. For example, the droneoperator can be placed into a sensory deprivation tank, and an intuitivecontrol of the drone can be accomplished using for example, thedetection of the position of outstretched arms of the user. As anexample, by controlling the rotation of the hand, such as one might dowhen driving down the road with the hand out the window, the wingcontrol surfaces can be remotely actuated to enable the operator tointuitively control the drone. Further, for entertainment, learning,therapeutic, military and/or other functional use, the operator can begiven a highly immersive illusion of real flight. Since the droneoperator is in the sensory deprivation tank, his or her brain will bemore receptive to the sensory cues that are applied. Thus, for example,a widescreen, or eyeglass, display can be used to provide visual cues.

FIG. 16 shows a user wearing a system for applying audio/visual/hapticcues and for receiving control intention input via electrical signalsreceived from the user's body. The user also wears a skullcapconstructed similar to the haptic sleeve and suit shown herein to mapand detect electrical signals received from the user's brain. There areapplications where full body haptic stimulation combined withsimultaneously applied sensory cues can be effective for learning,entertainment or rehabilitation. For example, exemplary embodiments canbe used as a rehabilitation device, to induce movement in the individualfingers on a hand or invoke involuntary movement of leg muscles. Thefull body haptic interface shown can be segmented depending on the need,and the resolution of the applied electrical signals can be as refinedor course as necessary. That is, for example, the muscles that controlmovement of each finger can be separately targeted.

As a non-limiting exemplary utilization, multiple sensory cues can besimultaneously received by a patient, such as a stroke victim. Forexample, audio (musical tones), visual (displayed hand position on akeyboard) and haptic (vibration applied to individual fingerscorresponding to notes being played) can be used to “teach” a patienthow to play a simple song on a piano keyboard. By providing thesimultaneously applied multiple sensory cues, the goal is to strengthenthe patient's brain and nervous functions that control hand movement. Inaddition, or as an alternative, to the vibration received by eachfinger, the electrical stimulation of the nerves that control theindividual finger movement can also be targeted. In accordance with anembodiment, the nerve stimulation is applied in a more general way(e.g., stimulate the middle and ring finger simultaneously) whileapplying the haptic sensation to only the individual targeted finger(e.g., the ring finger).

Another exemplary utilization is rehabilitation of a stroke victim orother brain injury or deficiency victim enabling more rapid rerouting orrewiring of the various communication signals between areas of thebrain. For example, if the portions of the brain related to auditoryprocessing are damaged or otherwise defective, the visual and sensorycues, along with the audio cues, generated to stimulate the variousprocessing centers of the brain of the stroke victim will help toreinforce newly learned auditory responses as the brain rewires thosespecific portions related to auditory processing. Another exemplaryutilization can be to enhance the rehabilitation of spinal cord and/ornerve damage patients. In this case, the haptic stimulation inconjunction with the auditory and visual stimulation or sensory cueswill enable a nerve and or spinal cord damaged patient to begin theassociation of the sense of touch with the audible and visual sensorycues, thereby strengthening the neural pathways that create either newmuscle memory or help repair damaged pathways and memory associations.

The first plurality of sensory cues may comprise visual sensory cues forproviding a virtual visual indication to the user of an event. Thevisual sensory cues may include video data mapped to at least one ofcolor and intensity of an image of the event. The haptic sensory cuescan be generated dependent on the mapped video data. In this case, themapped video data is converted by the microprocessor to correspondingcharacteristics of the computer controlled serially generated electricalsignals, enabling, for example, the visual sight of a firework explodingin midair being experienced as synchronistic haptic signals distributedover one or more body parts of the user, having an intensity and/orinvoking a sensation dependent on the visual characteristics over timeof the firework exploding in midair and fizzling out.

The sensory cues can be computer controlled using an algorithm thatgenerates signals capable of being perceived by a user, the plurality offirst sensory cue being serially generated and effective for stimulatingat least one sense of the user.

The first plurality of sensory cues may comprise auditory sensory cuesfor providing a virtual auditory indication to the user of an event. Theauditory sensory cues can include sound data mapped to stereo,multichannel and/or binaural channels. The haptic sensory cues aregenerated dependent on mapped sound data. In this case, the mapped sounddata is converted by the microprocessor to corresponding characteristicsof computer controlled serially generated electrical signals, enablingfor example, the audio sensation of the doppler shift of a moving trainwhistle being experienced as synchronistic haptic signals sweeping overtime across one or more body parts of the user, having an intensity orinvoking a sensation dependent on the audio characteristics over time ofthe doppler shifted train whistle getting louder, raising in pitch thengetting softer and lower in pitch as the sound fades out.

Time sequential first sensory data may be received from the remotetransmitter. A plurality of first sensory cues are generated capable ofbeing perceived by a user. The plurality of first sensory cue beingserially generated in synchronization dependent on the first sensorydata. The plurality of first sensory cues are effective for stimulatingat least one sense of the user. The haptic sensory cues are generated insynchronization dependent on the plurality of first sensory cues.

The time sequential data may include at least one sensed condition thatis sensed at a location remote from the user. The remote transmitter canbe part of a remotely controlled vehicle, such as a drone, robot orremote vehicle.

In accordance with a non-limiting exemplary embodiment, a human/machineinterface includes a plurality of conductive patches for applying anelectrical signal through the skin of a user to stimulate electricalsignal receptors. A signal generator generates a plurality of hapticcues in the form of electrical signals applied to the skin of the userthrough the plurality of conductive patches. The plurality of hapticsensory cues are capable of being perceived by a sense of touch ormuscle movement of the user.

The plurality of electrical signals have at least one characteristicincluding location, timing, pulse length, frequency and amplitudeeffective to cause at least one of a predetermined sensation and musclemovement in the user. The electrical signal receptors comprise at leastone of muscles, nerves and touch receptors. The signal generator mayalso generate a plurality of first sensory cues capable of beingperceived by a user. The plurality of first sensory cues are timesequentially generated and effective for stimulating at least one senseof the user. The plurality of haptic cues are time sequentiallygenerated in synchronization dependent on the time sequentiallygenerated plurality of first sensory cues. An x and y conductor gridprovides electrical communication of the plurality of electrical signalsfrom the signal generator to the conductive patches.

FIG. 17 shows a human/human interface where the haptic, visual and audioexperiences of one user is transferred to another user. The user alsowears a skullcap constructed as an EEG hairnet and along with the hapticsuit shown herein to map and detect electrical signals received from theuser's brain. In accordance with an exemplary utilization, the audio,visual and haptic data of another individual can be collected and usedto replicate for the user an experience perceived by the other. One usercan be outfitted to capture sensory information, for example, usingstereo video cameras located near the user's eyes and binauralmicrophones located near the user's ears. The electrical activity of theuser's muscles and nerves can be collected via a haptic suit havingconductive patches that amplify the received electrical signalscorresponding, for example, to muscle movement. This collected timesequential data (the data may change over time), for example, resultingin muscle contractions that change the position of the user's arm. Theother user receives the collected data and experiences the same audioand video information as experienced by the user. The other user alsohas involuntary muscle contractions caused by the collected haptic datathat results from the user's muscles contracting to change arm positionbeing applied to the other user causing similar change in the otheruser's arm position.

It is expected that the human/human interface can be used to simulate aninteraction between a user and a computer generated avatar. That is, inthis case the user is interacting with an avatar, not experiencing anevent from the perspective of a remote person. The computer generatedavatar is perceived by the other user as if existing within the virtual“world” created by the audio, visual and haptic systems describedherein. It is well know that the sense of smell can invoke strong memoryassociations, particularly with loved ones. A non-limiting applicationfor the human/human interface could be to replicate a deceased orremotely located relative. Artificial intelligence and stored datapertaining to the relative's personality and experiences can be usedalong with the applied sensory cues to construct an avatar having theappearance and mannerisms of the relative that is perceivable by theuser. If the inventive system is capable of providing a strong enoughexperience (including, for example, a scent of a childhood home), it isconceivable that, for example, an immersive experience can be computergenerated that enables a long dead grandfather to be consulted to givethe user advice during a virtual “visit”.

In accordance with the inventive accelerated learning system, augmentedreality is provided through the use of sensory cues, such as audio,visual and touch cues. These sensory cues pertain to an event or action.A combination of sensory cues can include various points of view orperspectives, created from data collected from a time sequential datasource such as sensors onboard remotely operated vehicles, or from anexperience of another human, or through artificial intelligence andcomputer simulation.

In accordance with an exemplary non-limiting embodiment of the inventivehuman/machine interface, the haptic sensory cues can be utilized alongwith the visual and/or audio sensory cues to create a new kind ofentertainment, whereby, a song or visual piece, such as a painting ormovie, can be utilized to create the pattern of sensory cues perceivableby the human through two or more senses, such as sight, hearing, touch,taste and smell. In accordance with other embodiments of the inventivehuman/machine interface, the haptic sensations can be applied to one ormore parts of the body, such as the legs, thighs, arms, ribs, torso,neck, head, etc.

For example, a drumbeat from a musical piece being listened to can beapplied as haptic sensations to the legs of the wearer, while the pianoperformance (such as that recorded as the teaching cues of the pianoperformer) can be applied as haptic sensations to the fingertips of theuse, while simultaneously displaying a visual scene with elements(colors, intensity) synchronized to the musical performance.

In accordance with an embodiment of the inventive human/machineinterface, the sensory cues can be utilized to provide rehabilitation toa victim of a brain injury or other brain damage or learningdysfunction. In this case, the various portions of the brain related tothe processing of sound, touch and vision can be controllably andsimultaneously stimulated so that a weakened brain sensory stimulationprocessing center can be strengthened or rewired through the support ofstronger brain sensory stimulation processing centers. For example, astroke victim with damage to right side of the brain may have a loss offunction in the motor control of the fingers of the left hand. In thiscase, the haptic sensory cues applied to the fingers of the left handprovide touch sensory stimulation to the damaged portions of the brain,while the corresponding visual and audio cues reinforce the re-learningor rewiring of the damaged portions of the brain through the touchsensory stimulation.

FIG. 18 shows an inventive human/machine interface composed of tactilefinger tips. The tactile finger tips can be constructed similar to thehaptic sleeve and suit shown herein to map, detect and apply electricalactivity at the user's fingers. FIG. 19 shows an inventive human/machineinterface comprising an orb having haptic and pressure active fingergrooves. FIG. 20 shows the inventive orb with high resolution haptic andpressure active finger grooves. FIG. 21 shows a block diagram of circuitcomponents of the inventive orb. FIG. 22 shows elements of a handapplied wireless haptic information transducer of the inventivehuman/machine interface. In accordance with this non-limiting exemplaryembodiment, transducers are provided for detecting and applyingelectrical signals to the fingers of the user. A hand operated orb caninclude finger grooves that receive each finger and are lined with thetransducers for applying and receiving electrical energy and othertactile stimulation (e.g., vibrations or pressure). The orb comprises ahousing that holds transducers, accelerometers, microprocessors,vibrators, gyros, and transmitters, etc., enabling the use of the orb asa human/machine interface such as a wireless three dimensional mouse orwireless joystick-like controller for gaming, entertainment, military,business, remote control and many other uses.

FIG. 23 illustrates an audio/visual/haptic signal collecting system.FIG. 24 illustrates an audio/visual/haptic signal applying system. FIG.23 schematically shows a system for collecting (recording, transmitting)haptic, auditory and visual information in accordance with the inventivehuman/machine interface. To record the audio, haptic and visualinformation during, for example, a non-limiting utilization of theinventive system during a piano session, finger position sensing glovescan be used with a digit/key detecting keyboard. The microphone is usedto record the notes played on the piano, the recording can be donebinaurally to enable more accurate immersion of the collected sensoryinformation. The user (performer) wears haptic signal detecting glovesso that the piano keys that are played, can be determined. Themicrophone simultaneously records the sounds generated by the piano whenthe piano keys are played. Further, a visual information recorder, suchas a video camera or specially constructed eyeglasses that include acamera, are used to record from the performers perspective, the hand andfinger positions of the performer while playing the piano. By thissystem, the experience of the piano player is collected from theperspective of three sensing cues: audio, visual, and haptic.

FIG. 24 schematically shows a system for applying the collected audio,visual and haptic information to a remote user (remote in time and orlocation). In accordance with the remote experience is achieved bysimultaneously stimulating the auditory, visual and haptic senses of auser, to simulate and/or augment an actual performance of an event, suchas the playing of a song on a musical instrument, for example, at apiano. Collected or artificially generated sensory cues are provided tothe user through an auditory information transducer, haptic informationtransducer and visual information transducer. The respective transducersare connected to and activate a corresponding interface device, such asheadphones, gloves and displays (for example, enabling the human/humaninterface shown in FIG. 17).

The inventive human/human interface can be used for acceleratedlearning, entertainment and other human sensory and cognitiveinteractions. For example, in the case of a haptic informationtransducer, a vibration buzzer (such as a piezo or motor drivenmechanical vibrator) and/or electrical signals can be applied to theindividual fingers and arm muscles and nerves of the user, for example,a student during a lesson learning session. In the case of the display,it may be, for example, specially constructed eyeglasses that displayvisual information that has been collected or artificially createdcorresponding to the learned event or entertainment session. Speciallyconstructed VR goggles or eyeglasses may display visual information asan overlay, picture in a picture, or other simultaneously displayedvideo information while the user also sees the real world imagery. Forexample, when learning to play the piano, the student may be sitting atthe piano and able to see a sheet of music and also see the piano keyswith his hand and finger positions in real time, while also seeingvisual sensory cues that is being generated and supplied to thespecially constructed eyeglasses. Also, the inventive human/humaninterface can be used for accelerated learning that takes place remotein time and/or location from the instrument or teacher, so that thestudent feels, hears and sees sensory cues corresponding to the learningof the event at any time and place remote from the instrument. Thisaccelerated learning system is designed to create associative memory inthe user corresponding to muscle memory (haptic information), auditorymemory (auditory information), and visual memory (visually displayinformation).

To record from the performers visual perspective, video recordingglasses such as Google glass, can be used. Visual and audio playbackwhen in lesson mode can be done using video glasses that includeheadphones. Haptic gloves are worn that include a buzzer or vibratorand/or electrical signal conductors for triggering sensation in eachfinger or selected muscles and nerves of the user. An LED can also belocated on each finger or located near each finger tip. For example, thestudent receives the visual cues as received during a remote learningsession at the instrument to create a learned visual memory of what thestudent visually experiences when seated at the piano. The inventiveaccelerated learning system obtains and reinforces the memoryassociations of the sensory cue whether at the piano or remote from theinstrument. Plus, the user is more able to reinforce the associate tomemories of the sensory cues that make up the performance of an event,such as the playing of a piece of music. In accordance with thisnon-limiting utilization of the inventive embodiments, the user wearshaptic stimulators on the tips of their fingers. The haptic stimulatorscan be, for example, small buzzers, the haptic stimulator can be amechanism that applies an electrical pulse directly or indirectly to themuscle or muscle groups of the user to cause a sensation or contractionin that muscle group that corresponds to a particular finger that is tobe used to play, for example, a key on the piano during the learningsession. For example, the memories associated with the playing of apiece of music, in accordance with an embodiment of the invention, willinclude audio, visual, and tactile (haptic or other stimulation) cuesthat are generated and that can be repeated over and over to instill theassociative memory that is built up during the course of conventionalmusic practice at an instrument.

In accordance with non-limiting exemplary utilizations of the inventiveembodiments, to further enhance the entertainment, remote control and/orlearning experience, chemicals released by the brain systems can bedetected from a user that is controlling a remote vehicle or learning apiece of music at a practice session at the instrument. As anotherexample, the brain activity of a student can be sensed using well-knownbrain scan techniques (such as those described in the background) andthe applied sensory cues can be the focus of the different brainactivities related to auditory, visual, and haptic sensory cueprocessing to further reinforce and enhance the learning experience. Thebrain activity (e.g., the human/human interface shown in FIG. 17) canalso be detected as part of the collected data shared between thehumans. This data that is specific to the human physiology of theindividual can be used to enhance the experience of the user and as partof a collected data base to further refine the contraction andutilization of the embodiments described herein. The inventiveembodiments, such as the human/human interface and accelerated learningsystem can be applied to other activities, including but not limited tosports, school work, performing arts, military exercises, video gaming,etc. As is also described herein, aspects of the non-limiting, exemplaryembodiments can be utilized for a number of different fields, includingentertainment, military, sports, video gaming, remote controlled robots,drones and vehicles, other musical instruments, etc.

FIG. 25(a) shows an artificial real-time perspective view of a UAV asdisplayed on a visual cue system of the inventive human/machineinterface. Data from onboard cameras plus onboard accelerometers, GPS,etc., plus stored image data of the drone are used to create real-timeartificial perspective of drone in flight that is received as the visualsensory cues. As illustrated, the pilot can perceive the visual image ofthe drone as if flying along side the drone (e.g., in formation with thedrone). Alternatively, the drone and the scene around the drone canappear to the pilot from any other visual perspective.

FIG. 25(b) shows a 360 degree camera system for collecting videoinformation onboard a remote machine, such as a drone.

In accordance with this non-limiting, exemplary embodiment the inventivehuman/machine interface (i.e., at the pilot's location) may bephysically located relatively nearby to the drone and receive the timesequential data from a remote transmitter on the drone using line ofsight wireless transmission. In this case, the collected time sequentialdata (e.g., audio, video and haptic signals) transmitted from the droneto the pilot and the flight control signals transmitted from the pilotto the drone are received at the pilot and the drone essentially inreal-time due to the light of sight wireless transmission. Alternatively(or additionally), the inventive human/machine interface (i.e., at thepilot's location) may be located relatively far from the drone andreceive the time sequential data from the remote transmitter on thedrone using relayed wireless transmission, such as via a satellite link.The collected time sequential data (e.g., audio, video and hapticsignals) transmitted from the drone to the pilot and the flight controlsignals transmitted from the pilot to the drone are received at thepilot and the drone with a delay caused the relayed wirelesstransmission.

At the location of the pilot, a plurality of haptic sensory cues aregenerated capable of being perceived by the pilot. The haptic sensorycues are received by the pilot as computer controlled serially generatedelectrical signals. The electrical signals invoke a perception by thepilot related to the sense of touch. These received haptic sensory cuescan be applied as computer controlled electrical signals that are mappedto the body of the pilot so that different body parts receive differentsensory stimulation. For example, the hands and arms of the pilot may beconsidered the human embodiment of the control surfaces, such as flapsof a drone plane. The feet and legs of the pilot may be considered thehuman embodiment of propulsion components, such as the engines of thedrone plane. In this example, the flexing of one or both feet of thepilot can be detected and converted to flight control signals to controlthe engine speed (and thereby control the speed of the drone). Enginespeed time sequential data received from the drone can be converted intoa haptic sensory cue that is displayed along with visual speed data,such as GPS determined speed relative to ground, so that the pilot hasan intuitive sense of the drone engine speed (for example, intensity ofa sensed vibration can be correlated with the RPM of the engine) andalong with the visual confirmation of the drone speed relative toground. In accordance with the inventive human/machine interface, thepilot receives multiple sensory cues that are inter-related andsynchronized to indicate the flight conditions of the remote drone.

The haptic sensory cues are generated and applied to the pilot insynchronization dependent on the time sequential data that is receivedfrom the remote drone. In addition to the time sequential data thatpertains to the haptic cues, time sequential first sensory data is alsoreceived from the remote transmitter. This time sequential first sensorydata may be, for example, video or audio data that is collected byappropriate components on the drone. A plurality of first sensory cuesare generated capable of being perceived by a pilot. The plurality offirst sensory cues are serially generated in synchronization dependenton the first sensory data. That is, for example, the sequential framesof a video displayed to the pilot replicate the visual informationcollected by cameras on the drone in time sequence. The plurality offirst sensory cues are effective for stimulating at least one additionalsense of the user, including vision, hearing, smell and taste (in thisexample, vision). The haptic sensory cues are generated insynchronization dependent on the plurality of first sensory cues. Thatis, the haptic sensory cues represent the flight conditions (e.g.,control surface orientation and air pressure, etc.) experienced by thedrone synchronized to the visual information from one or more cameras onthe drone. One or both of the time sequential data and the timesequential first sensory data may include at least one sensed conditionthat is sensed at a location remote from the user. The remotetransmitter can be part of a remotely controlled vehicle, such as adrone, robot or remote vehicle. This enables, for example, the pilot tointuitively “feel” the forces on the drone while visually seeing theresults of a flight maneuver of the drone, such as a banking turn. Thissensory feedback to the pilot's control of the flight enables the pilotto have an intimate and immersive perception of the drone's flight.

The 360 degree camera system collects video information onboard thedrone. The placement and number of cameras is effective to enable a fullsphere of views available to a pilot wearing, for example, a headtracking virtual reality headset, such as the Oculus Rift. Well knownsoftware and camera lens configurations can be used, for example, tostitch together the video feeds from the cameras so that a seamless ornearly seamless video presentation is available to the pilot. As thepilot looks right, left, up, down, for example, the movement of thepilot's head is tracked and an appropriate video scene can be generatedin 3D on the virtual reality video headset. Fewer cameras and theplacement of the cameras can be at convenient locations onboard thedrone as determined, for example, by the weight and cost constraints.For example, forward looking cameras can be disposed at the front of thedrone and rearward looking cameras disposed at the rear. The perspectiveand zoom of the camera image data can be controlled via software so thatthe pilot may experience the visual cues as if the pilot is physicallylocated anywhere on the drone (cockpit, tail, wingtips, etc.). Also, thecollected video data can be combined with computer generated images sothat the perspective viewed by the pilot can be from outside the drone.For example, the pilot can view the drone he or she is remotelycontrolling as if flying along side or behind the drone (e.g., theperspective shown in FIG. 25(a)).

Although this non-limiting exemplary embodiment describes haptic sensorycues combined with auditory and/or visual sensory cues, the combinationof sensory cues could be any combination of the senses perceivable by ahuman, including smell, taste, hearing, sight and touch.

FIG. 26 illustrates a chair configured to receive and apply haptic andaudio cues. The human/machine interface can be configured as acomfortable chair, for example, to allow a drone pilot to maintain along duration mission. The sensory cues can be generated by componentsthat are integral to many user-friendly structures. For example, thehaptic sensory cues can be mapped to a chair, bed, clothing or apparatusthat can be worn by the user. FIG. 26 illustrates a massage chair havingzones (shown as different shades of grey) corresponding to various bodyparts of user, with significant contact with large portions of the skinof the user and the weight of the user facilitating contact with theconductive patches (e.g. electrodes) used for applying and detectingelectrical activity. In addition, or alternatively, the zones can applyvibrations and other perceivable haptic sensations (such as from movingcomponents under the fabric or covering of the chair) to the body partsof the user. These chair haptic sensations can be used to providerelatively larger sensory perceptions (e.g., over larger surface areasof the body) while, for example, the x-y grid and conductive patchesapply electrical signals to smaller, more targeted areas. The x-y gridand conductive patches can apply the electrical signals using anelectrode scan technique that is similar to the driving of an activematrix display. The haptic signals can be applied to create sensationsperceivable by the user. The sensations can be created throughelectrical stimulation, and/or through vibrations, related toentertainment, learning, physical therapy, etc. For example, in the caseof a massage chair, a soothing massage can be applied wherein themassage to various parts of the body are mapped to the differentfrequencies of a piece of music. The sensory cues can also include othersenses, such as taste and smell. In this case, the senses of tasteand/or smell can be utilized to provide positive and negativereinforcement of a learned activity. For example, in the case of a droneoperator learning to determine how to recognize friend or foe, during atraining exercise a visual sighting that challenges the operator withmaking a correct snap determination of friend or foe can be reinforcedby providing a pleasant smell when a correct determination is made andan unpleasant smell when an incorrect determination is made. By thisapplication of additional sensory cues as reinforcement to learnedbehavior or responses, another processing center of the brain is broughtinto the combined sensory processing learning experience. The differentranges of music frequency can also be mapped to visual stimulation,applied for example, using a 3D VR headset, in the form of light colors.The light colors can correspond, for example, to the sensitivity of thehuman eye to color stimulation. Thus, for example, the color can begenerated by LED lights that match the peak wavelength sensitivity ofthe cones of the human eye. The three types of cones have peakwavelengths near 564-580 nm, 534-545 nm, and 420-440 nm, respectively.

FIG. 27 illustrates a visual sensory cue showing an actual tennis racketseen from the perspective of the user with an overlay of a virtualtennis ball generated using computer program code and displayed using anaugmented reality display, such as augmented reality eyeglasses. Thisrepresents another non-limiting exemplary application of the inventivehuman/machine interface. In this case, a body member of a user receiveshaptic information pertaining to a virtual event that uses a performanceobject controlled by the user. The body member can be part of the user'sbody, such as the arms and shoulders, and the event can be a sportingactivity, such as tennis. The performance object in this case would be atennis racket and the position of the performance object can be detectedby appropriate proximity sensor, motion detectors, tilt detectors, alaser positioning system, and other mechanisms used to detect theposition of an object in three-dimensional space. The performanceelement in this case may be the handle of the tennis racket, and itsposition relative to an arm of the user as a tennis ball approaches andis struck by the racket can be determined. The tennis ball can be anactual tennis ball, or a computer generated tennis ball that the usersees and reacts to during the collection of the sensory cues data thatwill be used to teach the performance. This mechanism and method fordetecting and recording (data collection) of the position of body partsand performance objects/performance elements is used to collect thesensory cues that are used to teach the event and build up memoryassociations of the event in the various processing centers of thestudent's brain. The body member that is detected during the recordingof the event performance and then stimulated during the learning lessonor entertainment session can be at least one of a finger, toe, hand,foot arm, leg, shoulder, head, ears and eyes of the user. This techniqueof using the inventive accelerated learning system can be used, forexample, to create a virtual sport video game. Similar alternatives canbe constructed for other events, such as controlling a remotelycontrollable system, for example, the flying of a drone airship, a spaceexploration probe, the playing of a guitar, the assembly of a weapon,entertainment or brain rehabilitation to help “rewire” the brain of astroke victim or brain damaged patient, other cognitive therapyincluding enhanced learning, or any other event where a user can benefitfrom recorded sensory cues that stimulate the various processing centersof the brain. The non-limiting embodiments can also be utilized, forexample, to provide muscle memory association and/or entertainment andrehabilitation using full body activities, such as martial arts, skiing,diving, etc.

FIG. 27 illustrates a visual sensory cue showing an actual tennis racketseen from the perspective of the user with an overlay of a virtualtennis ball generated using computer program code and displayed using anaugmented reality display, such as augmented reality eyeglasses.Eye-hand coordination for playing tennis can be taught using anembodiment of the inventive accelerated learning system. In this case,the visual sensory cue can be the tennis ball coming towards the user,and the head movement to bring the ball into the racket. The hapticsensory cues can be electrical impulses applied to the muscles of thearm to strike the ball with the racket. Also, impulses can't be providedto the muscles controlling head movement. Also, shoulder and backmovement, and various other muscles that are major factors inpositioning the racket to strike the ball.

FIG. 28(a) shows a UVS configured as a biomimicry bird at the start of apropulsion flap. FIG. 28(b) shows the UVS configured as the biomimicrybird on a upward stroke. FIG. 28(c) shows the UVS configured as thebiomimicry bird soaring. Biomimicry can be usefully applied both in thecreation of a highly responsive remote controlled drone, and to provideguidance to determine what type of and where to locate sensors thatprovide a haptic feedback experience to the user controlling the drone.For example, in the case of a drone constructed based on the biomimicryof a bird, flexure, rotation, angle and pressure sensors can be disposedat the general locations where on an actual birds body the detection ofthese forces are used by the bird in controlling flight. The pilot canreceive haptic information that is based on sensing the change inangles, flexure, rotation and pressures experienced at the joints andsurfaces of the UAV. Drones can be constructed using biomicicry, withsensors located at pressure and motion change (e.g., flexing, hinging)locations of the drone. The telemetry of the transducers/sensors can beused to provide haptic feed back to the pilot.

FIG. 29(a) shows a bird with a control and communication circuit fixedto its back. FIG. 29(b) shows a bird with the control and communicationcircuit blocking muscle signals from the brain of the bird and applyingcomputer controlled muscle signals to the flight muscles of the bird.FIG. 29(c) illustrates the skeleton and feathers of a wing of a birdhaving sensors and transducers for remote computer-controlled flight.The sensors can be placed at the location where the bird feels theflight conditions, such as at the fixation point of the feathers(particularly those used to sense and control flight). The electricalactivity generated at these fixations points can be detected and thatdata used by AI and/or computer manipulation to create an overall “flyby wire” type of control so that the human pilot does not necessarilyneed to know every detail of the data but rather is given, for example,the trends determined from the data that are needed to adequatelyprovide through the human/machine interface meaningful perceptions inthe pilot and control intentions from the pilot of the remote controlledflight of the bird.

In accordance with a non-limiting exemplary embodiment, a plurality ofhaptic sensory cues are generated capable of being perceived by a user.The plurality of haptic sensory cues are dependent on a determinedcondition of at least one movable member of a performing body performingan event. The plurality of haptic sensory cues are effective forstimulating a touch processing center of a brain of the user based onthe determined condition. A plurality of visual sensory cues aregenerated capable of being displayed to the user on a video displaydevice. The visual sensory cues provide a virtual visual indication tothe user of a position of at least one of the at least one moveablemember and the performing body. The visual sensory cues are effectivefor stimulating the visual processing center of the brain of the user.The visual sensory cues are synchronized with the haptic sensory cues sothat the position is virtually visually indicated in synchronizationwith the haptic sensory cues, and so that the visual processing centeris stimulated with the visual sensory cues in synchronization with thehaptic sensory cues stimulating the touch processing center.

The synchronized stimulation of the touch processing center and thevisual processing center can be used for teaching the user to perform aversion of the event. The synchronized stimulation of the touchprocessing center and the visual processing center can be used forenabling the user to remotely control the performing body performing theevent. The performing body may comprise a human, and the movable membermay comprise a body part of the human. The performing body may comprisean animal, and the moveable member may comprise a body part of theanimal. The performing body may comprise a remotely controlled movingobject, and the moveable member comprises a mechanical component of theremotely controlled moving object

The movable member can be a finger, the performing body can be a human,and the event can be playing a piece of music. The movable member can acontrol surface, the performing body can be a drone, and the event canbe flying the drone. The movable member can be a wing, the performingbody can be a bird, and the event can be the bird flying.

For example, by selectively disrupting signals from and to the brain ofthe bird, and instead applying computer generated electrical signals,the flight of a bird can be controlled as if flying a drone. Telemetrycollected from the bird, including an onboard camera and conditionssensors (e.g., a rotation transducer detecting the rotation of thebird's wing) can be used to indicate the flight characteristics of thebird to the pilot. The flight characteristics could be applied as asensed involuntary urging of the pilot's own arms replicating the bird'swing position, and/or the telemetry can be utilized by a computerprocessor to enable the pilot's acquisition of useful flight informationwithout overwhelming detail (similar to the fly-by-wire techniques of amodern jet fighter). A control and communication circuit carried by thebird can include a GPS system to enable, for example, an automatichoming feature that returns the bird safely to base or navigates to amission target autonomously and/or under pilot control.

FIG. 30(a) is a flow chart illustrating the steps for collecting datasets of a sequence of sensory activity of an event to be replicated,transmitted and/or recorded. FIG. 30(b) is a flow chart illustrating thesteps for generating data sets of a sequence of sensory activity of anevent that has been collected. FIG. 30(a) is a flow chart illustratingthe steps for collecting data sets of a sequence of sensory activity ofan event to be replicated, transmitted and/or recorded. The collecteddata can be from an actual events made from a real world action, such asfor controlling a drone flight or for human/human interaction, or thecollected data can be determined from a computer program code, or acombination of real world collected data and computer generated data, sothat data sets of a sequence of sensory activity can be generated duringa remote control activity, entertainment experience, and/or a learningsession. FIG. 30(b) is a flow chart illustrating the steps forgenerating data sets of a sequence of sensory activity of an event thathas been collected. The generated data can be from the collected data ofactual events made from a real world action, such as a piano key beingplayed, or the generated data can be determined from a computer programcode so that data sets of a sequence of sensory activity can begenerated.

FIG. 31(a) is a perspective view showing an embodiment of a signalelectrode having conductive bumps. FIG. 31(b) is a cross section ofmid-forearm showing conductive bump signal electrodes selectivelyapplying and detecting electrical activity to muscles and nerves. Inaccordance with an exemplary non-limiting embodiment, the electricalsignals applied and received from the user can be applied/received viasignal electrodes constructed having conductive bumps. The conductivebumps enable the electrical activity to be transmitted through thedirect contact with the skin. Conventionally, for example, disposablegel electrodes are used to apply and/or receive electrical activity forexample, in EMG applications. However, the application of gel electrodesis time consuming and inconvenient, especially if there is hair on theuser's body part on which the electrode is being applied. In accordancewith a non-limiting embodiment, addressable conductive bumps can be usedto apply and receive the electrical activity targeted at specificmuscles and nerves of the user. To maintain good electricalcommunication with the skin, a compression sleeve can be used to urgethe conductive bumps towards the skin of the user. Similar to theconductive patch human/machine interface described here, the conductivebumps can be individually addressable via a suitable conductive pathwayfrom a signal generator (e.g., microprocessor signal control circuit).This enables, for example, specific targeting of physiological features,such as individual nerves, or larger areas of electrical stimulation,such as portions of muscles. A plurality of conductive patches orconductive bumps apply an electrical signal through the skin of a userto stimulate electrical signal receptors. A signal generator generates aplurality of haptic cues in the form of electrical signals applied tothe skin of the user through the plurality of conductive patches. Theplurality of haptic sensory cues are capable of being perceived by asense of touch or muscle movement of the user. The plurality ofelectrical signals may have at least one characteristic includinglocation, timing, pulse length, frequency and amplitude effective tocause at least one of a predetermined sensation and muscle movement inthe user. The electrical signal receptors comprise at least one ofmuscles, nerves and touch receptors. The signal generator furthergenerates a plurality of first sensory cues capable of being perceivedby a user, the plurality of first sensory cue being time sequentiallygenerated and effective for stimulating at least one sense of the user.The plurality of haptic cues are time sequentially generated insynchronization dependent on the time sequentially generated pluralityof first sensory cues. An x and y conductor grid can provide electricalcommunication of the plurality of electrical signals from the signalgenerator to the conductive patches.

The somatosensory system of the human body is a complex sensory system.It is made up of a number of different receptors, includingthermoreceptors, photoreceptors, mechanoreceptors and chemoreceptors. Italso comprises essential processing centres, or sensory modalities, suchas proprioception, mechanoreception (touch), thermoception(temperature), and nociception (pain). The sensory receptors cover theskin and epithelial tissues, skeletal muscles 18, bones and joints,internal organs, and the cardiovascular system.

In accordance with an aspect of the invention, a plurality of hapticsensory cues are generated capable of being perceived by a user 12. Thehaptic sensory cues are received by the user 12 as computer controlledserially generated electrical signals. The electrical signals invoke atleast one of a involuntary body part movement and a perception by theuser 12. The involuntary body part movement causing at least an urgingtowards at least one of a predetermined motion and a predeterminedposition of the body part dependent on the computer controlled seriallygenerated electrical signals. The perception by the user 12 having apredetermined somatosensory sensation dependent on the computercontrolled serially generated electrical signals.

The haptic sensory cues may invoke the perception by stimulating asomatosensory system of a user 12 comprising at least one receptorincluding thermoreceptors, photoreceptors, mechanoreceptors andchemoreceptors to cause the user 12 to perceive an experience of atleast one of proprioception (e.g., body part position and strength ofmovement), mechanoreception (e.g., touch), thermoception (e.g.,temperature), and nociception (e.g., pain).

The HHMI opens new avenues in human-automation interaction and control,including impacting the areas of accelerated learning, physical trainingand rehabilitation. The ability to identify muscle groups 18 at asufficient level of definition, and the ability to apply electricalsignals at a similar level, results in a system in whichpreviously-known actions and muscle movements could be developed forimproved physical training and correction of physical motion. Musclememory associated with nearly all kinds of human activities can be morequickly developed to learn, for example, a musical instrument or sporttechnique. For military applications, rapid muscle memory build up couldenhance the training of soldiers in basic and advanced weapons.Additionally, new forms of safety restraints could be provided in whichthe human user 12 is prevented through the HHMI-applied electricalsignals from taking an action that may result in injury or undesiredaction.

Medical use examples include non-invasive, non-chemical means tocounteract involuntary tremors caused by Parkinson's disease; strokeinjury and other brain damage rehabilitation through rewiring of thedamaged brain by the synchronized application of computer-controlledhaptic, audio and visual cues; and, the treatment of autism by providinga sensation replicating light pressure thereby providing therapeuticbenefits using a custom-calibrated, mobile and convenient system.

As shown in FIGS. 32 and 33, a non-limiting exemplary embodiment of aninventive haptic human/machine interface (HHMI) is configured as asleeve that can be worn by a user 12, with the detection and applicationof electrical signal activity obtained through a user-calibrated grid ofconductive patches or electrodes 14. FIG. 32 illustrates a user's arm 16without skin showing the relative locations of the muscle groups 18 ofthe arm 16. FIG. 33 illustrates the arm 16 with the HHMI sleeve havingelectrodes 14 targeting individual muscles 18 or muscle groups 18. TheHHMI sleeve may include an x-y grid of relatively smaller signalreceiving transducers or electrodes 14 and relatively larger signalapplying electrodes 14 targeting individual muscles 18 or muscle groups18 (see, for example, FIG. 72), or as shown I FIG. 33 the electrodes 14may be uniform in size and distribution. The HHMI may be in the form ofa comfortable, easily worn garment that is worn with little restrictionof movement.

Electrical signals are applied to the user 12 via the plurality ofelectrodes 14. Each electrode is disposable in electrical communicationwith one or more biological components of the user 12. At least oneelectrode is individually addressable to be selectively in an on-stateor an off-state. In the on-state the electrical signals flow through theelectrode to at least one biological component of the user 12. In theoff-state the electrical signals do not flow through the electrode tothe biological component. Each electrode is individually addressable todetect electrical activity of the biological component during a signaldetecting operation and apply the electrical signals to the biologicalcomponent during a signal applying operation.

The HHMI may be configured as a sleeve that is part of a garment, or aself-contained wearable electronic that maps the sources of electricalactivity (the subcutaneous muscles 18 and nerves) and hence determinesthe best locations to detect and to apply the electrical signals for aparticular user 12.

Neuromuscular electrical stimulation is applied as a low frequency,relatively high intensity pulse. The pulse, which may be biphasic,triggers the alpha motor nerves that cause muscle movement. The higherthe intensity of the electrical stimulus, the more muscle fibers will beexcited, resulting in a stronger contraction. The contraction can havedifferent speeds and duration of the contraction dependent on thecharacteristics of the applied electrical signal. The characteristics ofthe applied electrical signal can be controlled to cause isometricand/or isotonic muscle contraction, where an isometric musclecontraction leads to a tension in a muscle, without changing the lengthof the muscle, and isotonic muscle contraction, results in a shorteningof the muscle. In accordance with the inventive haptic interface, acomputer controls the characteristics of electrical signals applied to,for example, the motor neurons of the user's nervous system to cause adesired sensation and/or muscle movement. Exciting the motor neurons viathe body's nervous system produces a similar result as when the neuronsare excited through the computer controlled electrical stimulation.

In accordance with the inventive haptic interface, computer controlledelectrical signals can be applied with signal characteristics effectiveto stimulate one or more of the tactile receptors found in the skin. Thesignal characteristics are controlled to selectively stimulate thereceptors that have, for example, different receptive fields (1-1000mm2) and frequency ranges (0.4-800 Hz). For example, broadreceptive-field receptors like the Pacinian corpuscle produce vibrationtickle sensations. Small field receptors such as the Merkel's cells,produce pressure sensations.

The HHMI may be used for applications including accelerated learning,brain damage rehabilitation, military and sports training anddrone/robotic remote control and sensing. In some configurations, theHHMI includes a thin, flexible sleeve that is unobtrusively worn by apatient. The sleeve has many small electrodes 14 in contact with theskin surface, connected in a matrix and addressed, for example, usingelectronic techniques borrowed from active and passive matrix videodisplays. A lightweight, comfortable, haptic sleeve can be configuredhaving electrode size and density enabling automatic calibration to theunique physiology of the patient. The haptic sleeve provides preciseelectrical activity detection (for example, to detect the muscles 18 andnerves involved in even subtle arm movement indicating the onset of apersistent Parkinsonian tremor) and nearly instantaneous electricalsignal application (to cause involuntary and accurate muscle and nerveimpulses that counteract and negate the undesirable arm trembling thatwould have otherwise occurred). The applied electrical signal andresultant muscle contraction is perceived as a massage sensation by thepatient. In this case, the use of the HHMI provides a wearableelectronic device used as a non-invasive, non-chemical means toeffectively mitigate tremors caused. For example by stroke, accident orby Parkinson's disease.

In another human/machine interface example, a drone (or robot) isinterfaced with an operator via the HHMI and AR or VR components tocreate a remotality (a remote reality) for a pilot or operator, in asense making him feel as if he actually becomes the drone, having thesensation (sights, sounds, skin sensations) of flying like supermanwhile piloting a real-world flying drone.

In accordance with another aspect of the invention, a plurality ofhaptic sensory cues are generated capable of being perceived by a user12. The haptic sensory cues are received by the user 12 as computercontrolled serially generated electrical signals. The electrical signalsinvoke an involuntary body part movement causing at least an urgingtowards at least one of a predetermined motion. Alternatively, oradditionally, the signals invoke a perception by the user 12 having apredetermined somatosensory sensation dependent on the computercontrolled serially generated electrical signals.

FIG. 34 schematically illustrates an electrode equivalent electroniccircuit for applying and detected electrical signals. In an example useof the inventive HHMI, the exemplary embodiments include electricalcircuits, such as those shown here or equivalents, that are used todetect the onset of an involuntary tremor of a user 12. Electricalsignals are determined by a microprocessor to have electricalcharacteristics effective to mitigate the involuntary tremor. Theelectrical signals are applied to the user 12. The electrical signalsmay be applied to the user 12 via a plurality of electrodes 14 whereeach electrode is disposable, for example, using the haptic sleeve,garment or body suit shown herein. Each electrode is in electricalcommunication with one or more biological components of the user 12,such as the skin of the user 12 and through the skin the nerves andmuscles 18. As an alternative, or in addition to skin surface contact,one or more of the electrodes 14 may be disposed subcutaneously, forexample, to apply or detect electrical signal at muscles 18 or nervesthat are deep beneath the skin layer. These subcutaneous electrodes 14may be permanently or semipermanently left in place, or they may be, forexample, acupuncture-type needles that are applied and then removed whennot in use.

At least one electrode may be individually addressable to be selectivelyin an on-state or an off-state. In the electrical equivalent electroniccircuit switches 20 are symbolically shown. In an actual circuit, theon/off state can be controlled through electronic switch mechanisms thatinclude, but are not limited to transistors 22, reed switches 20,relays, optoisolators, and the like. A combination of known electricalcircuit components and microprocessor controlled devices can be used. Inthe on-state the electrical signals flow through the at least oneelectrode to at least one biological component of the user 12 and in theoff-state the electrical signals do not flow through the at least oneelectrode to the at least one biological component of the user 12.

FIG. 35 illustrates pulse square waves depicting computer generatedelectrical signals that can be selectively applied via the electrodeequivalent electronic circuit. The electrical signal (e.g., a hapticelectrical signal) may have characteristics that depend on a pulse wave,although the application of any electrical signal that results in adesired movement, urging, sensation, etc., may be applied. FIG. 36 showsthe electrode equivalent electronic circuit applying a selectedelectrical signal applied to selected electrodes 14. The HHMI can beconfigured to allow biphasic, multiphasic or monophasic actionpotential, with the electrodes 14 each individually addressable to beselectively on/off and allow positive or negative current flow throughthe individual electrode. This provides great flexibility to theelectrical signal application. The plurality of electrodes 14 areindividually addressable or can be addressed in clusters, and can beselectively grouped to form an electrode pattern that conforms to apreferred area shape, for example, to match the shape of the targetedmuscle or biological feature. The duration and frequency of each appliedsignal can vary among two or more signal choices. The applied signalscan be as complex as necessary and the applied location on the user'sbody can be as precise are required, to create involuntary fine motorcontrol to enable, for example, the finger patterns for actuatingmusical instrument keys 34 to be achieved either totally involuntarilyor aided and guided.

The electrode may be individually addressable so that when in theon-state a direction of current flow of the applied electrical signalscan be selectively at least one of positive or negative. The biologicalcomponent may comprises a component of at least one of a muscle,nervous, lymphatic, organ, skin, sensory and other biological system ofthe user 12. The electrode may be individually addressable in accordancewith pulse width modulation so that the effective electrical energy ofthe applied electrical signals flowing through the at least oneelectrode to the biological component can be independently reducedrelative to the applied electrical signals without pulse widthmodulation. The response of the muscle and nerves will tend to integratean applied pulse electrical signal.

Another electrode of the plurality of electrodes 14 may be individuallyaddressable in accordance with pulse width modulation so that theeffective electrical energy of the applied electrical signals flowingthrough the other electrode to the biological component is differentthan the effective electrical energy of the applied electrical signalsflowing through the first electrode to the biological component. Thisenables different areas of the biological component to receive differenteffective electrical energies of the same applied electrical signals. Aportion of the plurality of electrodes 14 may be selectively driven asgroups forming an electrode pattern conforming to a target area of thebiological component.

FIG. 37 illustrates an electronic circuit including a plurality ofaddressable electrodes 14 for applying and/or detecting electricalsignals to muscle fibers and nerves. Transistors can be used to switch ahaptic signal under the control of a controller. The controller is inturn controlled by a microprocessor. The control and microprocessor canbe integrated together, or separated elements. For example, themicroprocessor can be a smart phone or other readily availableelectronic device, or it can be a dedicated device. The controller maybe a small integrated circuit device that is associated with anelectrode or group of electrodes 14 and disposed within flexible circuitlayers of the HHMI. The electrical signals may be applied as hapticsensory cues received by the user 12 as computer controlled seriallygenerated electrical signals. The electrical signals invoke may invokeat least one of an involuntary body part movement having a predeterminedmotion dependent on the computer controlled serially generatedelectrical signals and a perception having a predetermined touchsensation dependent on the computer controlled serially generatedelectrical signals.

Exemplary schematic wiring diagrams of a driving circuit is shown inFIGS. 37-41. There are a variety of electronic circuits that can beemployed to create some or all of the features shown in the schematicwiring diagram. As the wiring diagrams illustrate, each electrode may beselectively addressed to be in an on or off state. The HHMI includes aplurality of electrodes 14 wherein each electrode is disposable inelectrical communication with one or more biological components of theuser 12, at least one electrode being individually addressable to be inan on or off state, wherein in the on state the electrode allows thehaptic electrical signal to flow to the biological component and the offstate does not allow the electrical signal to flow to the biologicalcomponent, the at least one electrode being individually addressable sothat when in the on state a direction of current flow of the hapticsignal can be either positive or negative through the at least oneelectrode. The biological component can comprises a component of atleast one of a muscle, nervous, lymphatic, organ, skin, sensory andother biological system of the user 12. That is, the biologicalcomponent is a system of the user's body that is reactive to the appliedelectrical signal. For example, in the case of the muscle and nervoussystems, the applied electrical signal can invoke one or both of aperceived sensation and an involuntary movement. The perceived sensationand/or involuntary movement can be felt by the user 12 as a guidingforce, for example, to urge the user 12 hands and fingers towards aposition of a pattern of a musical chord being played on a piano, with aselected individual finger sensation of striking a key of the pianoindicating the finger pattern and keys 34 that a struck by the fingersto achieve the desired musical chord.

FIG. 38 illustrates another electronic circuit including a plurality ofaddressable electrodes 14 for applying and/or detecting electricalsignals to muscle fibers and nerves. In this exemplary embodiment, atransistor 22 associated with an electrode can be used to allow currentflow in one direction and another transistor 22 can be used to allowcurrent flow in the other direction. At least one electrode isindividually addressable so that when in the on-state a direction ofcurrent flow of the applied electrical signals can be selectively eitherpositive or negative. In a medical use example, the onset of aninvoluntary tremor motion is detected in a body part (e.g., aParkinsonian arm/hand tremor) by amplifying the electrical activity inthe muscles 18 and nerves. This detected electrical activity is thenused to determine the characteristics of an electrical signal that isthen applied back to the muscles 18 and nerves to mitigate the tremormotion. The electrical signals are detected and transmitted throughsurface contact with the skin, the product is a wireless, wearableelectronic, with no chemicals or invasive and dangerous procedures.

FIG. 39 is a schematic of another electronic circuit example having aPWM driver for selectively applying a pulse width modulated AC and/or DChaptic electrical signal to selected addressable electrodes 14.Optoisolators can be used to separate, for example, a low voltage DCcontrol circuit from, for example, a high voltage AC haptic signal. Thecircuit may be constructed similar to the use of an integrated driverknown as WS2812 available from World Semi, China, for driving an RGB LEDarray, but adapted so that each electrode pair or selected grouping ofelectrodes 14 (e.g., ELECTRODE A and ELECTRODE 1 pair shown in FIG. 7,or electrode grouping comprising any two or more electrodes) can beactivated in a pulse width manner to selectively apply the hapticelectrical signal wherein an electrical circuit that includes theelectrode pair also includes a biological component such as the skin,nerves and muscles 18 of the user 12. Using a driver, such as WS2812also provides an advantage in that software and circuit devices, such asthe Arduino, can be readily adapted for the HHMI use speeding thedevelopment and providing the potential for open source advancements.The electrical circuits may include regulators to ensure that theelectrical signal applied is always within a safety constraint. Asanother similar example driver, the WS2811 8-bit PWM driver controlsthree LED (RGB) channels (total of 24 bits) and provides a potentialintegrated circuit that has a construction and functionality useful forillustrating some of the inventive concepts of the electrical circuitsshown herein. The use of these example drivers is for illustrativepurposes, there being other discrete electronic and integrated solutionsthat could be used.

Like persistence of vision, the detected and applied signals can besamples that are representative of muscle activity/detection and pulsesthat are effective to cause precise involuntary muscle pulses thatappear smooth. The applied signal can be as complex as necessary sothat, for example, a varying PWM pulse can be applied at varyingeffective strengths to nearly instantaneously varying locations andsurface areas of the user's skin.

The haptic sensory cues may stimulate a somatosensory system of a user12 comprising at least one receptor including thermoreceptors,photoreceptors, mechanoreceptors and chemoreceptors to cause the user 12to perceive an experience of at least one of proprioception,mechanoreception, thermoception, and nociception. The haptic sensorycues may be generated in synchronization dependent on time sequentialdata. The electrical signals simultaneously stimulate both theinvoluntary body part movement and the perception by the user 12 relatedto the sense of touch.

FIG. 40 is a schematic showing the electronic circuit example forapplying the electrical signal through muscle and nerve fibers through aplurality of individually addressable electrodes 14. In this circuit,the circuit module shown in FIG. 39 is reproduced for multipleelectrodes 14. The number of electrodes 14 can be significant, inparticular for a whole body HHMI or HHMI shirt. The driver andelectronic circuitry used to control a large number electrodes 14 mayborrow from, for example, known electronic circuit for driving passiveor active matrix displays, multiplexers, etc., but adapted as necessaryto apply the haptic electrical signal.

FIG. 41 is a schematic showing a repeatable circuit element forindividually addressing a respective electrode and a correspondingelectrode of a plurality of electrodes 14 to selectively apply, detector switch off signals to the addressable electrodes 14. Further, theleast one electrode can be individually addressable to provide arelative strength and/or duration of the applied signals that flowthrough the biological component through the corresponding electrode.For example, the electrodes 14 can be individually addressable inaccordance with pulse width modulation so that the effective electricalenergy of the applied electrical signals flowing through the at leastone electrode to the biological component can be independently reducedrelative to the applied electrical signals without pulse widthmodulation. Using this driving technique, haptic electrical signals canbe applied having a complex electrical characteristic having varyingeffective electrical energy applied as pulses at specific electrodes 14to cause precise movements and perceived sensations. The same electrodes14 can be used to apply the electrical signal generated by thecontroller or microprocessor and to detect the myographic data. Themicroprocessor controls the electronic circuit so that the hapticsignals are selectively applied to the electrodes 14, and the myographicdata are selectively detected from the same electrodes 14.

FIG. 42 illustrate an exemplary embodiment including an electroniccircuit for detecting electrical activity of muscles 18 and nerves froma plurality of electrodes 14. In accordance with another aspect of theinvention, electrical activity is received from at least one of muscles18 and nerves of a user 12. An electrical signal is determined havingcharacteristics based on the received electrical activity. Theelectrical signal is generated and applied to the user 12 to cause anaction dependent on the received electrical activity.

For example, in a medical use application, the received electricalactivity may be the result of an involuntary tremor of the user 12. Thecharacteristics of the electrical signal can be determined based on theinvoluntary tremor to cause involuntary muscle movement that counteractsthe involuntary tremor. The electrical signal can be generated as anelectrical signal having the characteristic to cause the involuntarymuscle movement that counteracts the involuntary tremor. The electricalsignal may be selectively applied to the user 12 using the addressableelectrodes 14 to cause the muscle movements that counteract theinvoluntary tremor. The electrical signal may be applied through aplurality of conductive patches (electrodes 14) for applying anelectrical signal through the skin of a user 12 to stimulate electricalsignal receptors. The applied electrical signal may comprise a pluralityof different electrical signals applied to different locations of theuser 12 via the circuit construction described herein so that thedifferent electrical signals have at least one varying characteristicincluding location, timing, pulse length, frequency and amplitudeeffective to cause at least one of a predetermined sensation and musclemovement in the user 12. The applied electrical signal can thus bereceived by electrical signal receptors of the user 12 comprising atleast one of muscles 18, nerves and touch receptors, causing aninvoluntary movement and/or a perceived perception of a somatic and/orkinesthetic sensation.

As discussed in more detail herein, in addition to the haptic sensorycues applied using the HHMI, a plurality of first sensory cues (e.g.,visual and/or auditory) can be generated capable of being perceived by auser 12. The plurality of first sensory cues are time-sequentiallygenerated and effective for stimulating at least one sense of the user12. The electrical signal may comprise a plurality of haptic cuestime-sequentially generated in synchronization dependent on thetime-sequentially generated plurality of first sensory cues. Theelectrical signal may be applied to the user 12 using a conductor gridfor electrical communication of the electrical signal to the conductivepatches.

The HHMI has many small electrodes 14 that are individually addressableto form localized groups conforming to the correct location and size ofthe patient's body to optimally apply precisely targeted electricalsignals and control subtle movement, such as finger, arm and handmovements. As shown the same electrodes 14 that apply the computergenerated signal, using a different addressing scheme that may includeground electrodes 14 positioned at bony parts of the arm 16, are locallygroup by the driving circuitry and software to form detection regions atisolated muscles 18 and nerves.

In accordance with an exemplary embodiment, the HHMI is configured as atherapeutic wearable electronic device that interfaces the user 12 witha small, mobile microprocessor, portable communication device, smartphone, tablet or the like. The HHMI includes electrodes 14 in contactwith the skin surface of the user 12, connected via conductive leads andindividually addressable. In accordance with exemplary embodiments, thesame electrodes 14 are used to detect and apply electrical signalsfrom/to the muscles 18 and nerves.

For example, the HHMI can be configured as a wearable electronic devicethat mitigates the effects of Parkinson's tremors without drugs orinvasive surgery. As described in more detail herein, there are alsoseveral notable other medical uses that start with this wearableelectronic device as an enabling technology for stroke and brain damagerehabilitation through the simultaneous controlled application ofsensory cues (for “re-wiring” a damaged brain).

In accordance with this exemplary use, the patient wears a comfortablegarment, like a long underwear sleeve, that creates a feedback loop fromthe involuntary tremor movement just as the body part e.g., arm/hand)begins to move (that is, the Parkinsonian shaking action caused byinvoluntary muscle contractions/relaxations). The feedback is in theform of the applied electrical signal that causes an opposing musclecontraction/relaxation (or that disrupts the nerve signals causing theinvoluntary muscle movement) that steadies the shaking arm 16 and hand.The patient wears this wireless, comfortable, haptic sleeve and feelsthis feedback as a gentle massaging action that pulses in sync with theinvoluntary tremor. But, the pulses are timed so as to prevent thetremor movement, and the undesirable shaking action is mitigated.

By selecting the appropriate individually addressable electrodes 14, thepattern of the electrodes 14 is shaped to match the underlying muscles18 and nerves. The HHMI may include registration indications, such as aseam that runs lengthwise, IR reflectors, or other indicia, and isoriented to the elbow and wrist of the user 12. The HHMI starts with aclose approximation because of alignment and generally consistent humanphysiology, after the calibration process the HHMI ends up withuser-specific and accurate mapping of electrodes 14 to the tremorcausing muscles 18 and nerves (activity detection) and the tremorcounter-acting muscles 18 and nerves (signal application).

Electrical signals from oscillatory body movement are detected todetermine the onset of tremor and to detect the muscles 18 that areinvoluntarily contracting/flexing to cause unwanted movement.Counter-tremor muscles 18 are determined that when stimulated willresist the unwanted movement. The strength and other characteristics ofan applied electrical signal are determined to counteract tremor andhold the body part substantially steady or at least mitigate the tremormotion.

FIG. 43 shows the muscles 18 and bones of the forearm 16 and hand of auser 12 that are used in hand/digits extension. The flexor digitorumprofundus 24 is used to bring the fingers in towards the palm and theextensor digitorum 26 is used to bring the fingers back away from thepalm. The ulna 28 is the bone that terminates at the elbow. In thisexample, a flexion-extension tremor causes an involuntary shaking of thehand. FIG. 44 shows the forearm 16 and hand of the user 12 illustratingan exemplary embodiment of the inventive HHMI, the electrodes 14 areshown without the housing or covering, flexible electronics, insulators,etc. so that the locations of the individually addressable electrodes 14with relation to the user's body anatomy can be more clearly understood.In actual construction, the HHMI is a multilayered structure that hasself-contained flexible materials and electronics. FIG. 45 illustratesdetected electrical signals indicating the onset of an extension portionof the tremor. The electrical activity causing the contraction of theextensor digitorum 26 is detected using the individually addressableelectrodes 14 that are in best electrical communication with theextensor digitorum 26. FIG. 46 shows the locations of the muscles 18having the detected and applied electrical signals. MRI or other imagingcan be used to get graphical starting point for electrode placement, thesleeve can include markers that align with indicators that naturallyoccurring such as elbow bone, and/or artificial and applied to the skinsurface, such as disks that are opaque to the imaging process andadhered to the skin. Once the imaging indicates where the muscles 18are, the location of the electrodes 14 can be biased for these areas asthe starting point of calibration or in lieu of calibration.

As an example, each of the individually addressable electrodes 14 can bescanned to see if electrical activity is occurring at the area of thescanned electrode. A reference electrode (or group of electrodes 14)Eref 30 may be located at the elbow joint, where the bone located closeto the skin surface acts as a neutral tissue for the detection ofelectrical activity using the electrodes 14 located in proximity to theextensor digitorum 26. The detected electrical activity at the groupingof detection electrodes Edet 32 is processed by the microprocessor todetermine if the activity is the result of a voluntary or involuntarymuscle and/or nerve action. For example, if the patient is known to havea flexion-extention tremor causing the involuntary shaking of the hand,then the microprocessor may be programmed to look for electricalactivity consistent with the tremor. If the detected electrical activityis indicative of the onset of a tremor motion, then the muscles 18 (inthis case, including the flexor digitorum profundus 24) that mitigatethe tremor motion can be determined and the appropriate counteractingelectrical signal can also be determined. The timing of thecounteracting electrical signals and other signal characteristics aredetermined by the microprocessor, for example, from data stored in lookup tables or using calculations based on measured responses of apopulation of humans to similar electrical stimulation and thanextrapolating from this data, for example using calibration data and thedetected activity, the electrical signals that will best mitigate thetremor. The individually addressable electrodes 14 that correspond tothe flexor digitorum profundus 24 may be determined, for example, from astored mapping of the electrodes 14 that is obtained during thecalibration and/or refinement procedures. The microprocessor controlsthe application of the counteracting electrical signals to tremormitigation muscles 18 using the electronic circuits described herein, orother appropriate electronic circuit (for example, a scanning ornon-scanning multiplexed circuit). The electrical signals that areapplied may be complex and different signal wave forms, effective signalstrengths, and the like may be used to apply multiple signals throughthe application electrodes Eapp 33 in proximity to the flexor digitorumprofundus 24. A kill switch (not shown) may be provided on the HHMI sothat the user 12 can easily stop the application of a computer-generatedelectrical signal.

The HHMI uses sensory feedback and algorithms to correctly control thecharacteristics of the electrical stimulation so that involuntary musclecontractions are induced that result in the movement or urging of thebody part of the user 12 (e.g., fingers) as if performed voluntarily bythe user 12 using precisely controlled muscle movements.

An example detection and application of electrical signals is describedherein, electrical signals can be detected and applied to otherbiological systems of the user 12 using the inventive HHMI. Thebiological system may include at least one of the musculatory, nervous,lymphatic, respiratory, circulatory, digestive systems, and other bodysystems that is reactive to electrical stimulation.

In some uses, such as accelerated learning or sports training, the HHMIcan be a component for detecting and applying haptic sensory data in anAR or VR system where a time sequential first sensory data may bereceived from a transmitter. A plurality of first sensory cues may begenerated dependent on the first sensory data and capable of beingperceived by the user 12. The plurality of first sensory cues may beeffective for stimulating at least one sense of the user 12. The hapticsensory cues can be generated in synchronization dependent on theplurality of first sensory cues. At least one of the first and thehaptic sensory data may include at least one sensed condition sensed ata location remote from the user 12. At least one of the first and thehaptic sensory data may include at least one pre-recorded sensedcondition. The sensed condition may be recorded as an audio signal, avideo signal, and/or data representing at least one of a position of abody part, a smell, a somatic sensation, a kinematic sensation, amovement, a pressure, a temperature, a time duration, an emotion, ataste, and a resistance.

In accordance with an aspect of the invention, a thin, flexible,conformable electrode is provided that provides face-to-face electricalcontact with the skin of the user 12, while being particularly adaptablefor use in the HHMI.

The HHMI may be provided as a wearable housing supporting the apparatusto provide a user-wearable electronic device. The wearable housing maycomprise a multilayered flexible electronic circuit including anelectrode layer comprised of a plurality of electrodes 14 having aconductive face disposed for making electrical contact with a biologicalsystem of the user 12 and at least one additional layer including atleast one of an electrical circuit layer, an electrical insulatinglayer, an electrical conducting layer, and a flexible covering.

FIG. 47 is a close up cross section showing an embodiment of anelectrode for use with the inventive HHMI having conductive loops. FIG.49 is a cross section showing the embodiment of an electrode for usewith the inventive HHMI having conductive loops. The electrode includesa plurality of conductive elements such as conductive loops that may beor may resemble, for example, a conductive hook and loop fastener suchas Velcro.

FIG. 48 is a close up cross section showing an embodiment of anelectrode for use with the inventive HHMI having conductive stems. FIG.50 is a cross section showing the embodiment of an electrode for usewith the inventive HHMI having the plurality of conductive elements asconductive stems. The stems may be formed, for example, from an expandedmetal, laser cutting, or stamping manufacturing process. The electrodemay comprise the individual stems formed as a thin, flexible hangingchads fixed at one end to a common sheet from which they are stamped.

FIG. 51 is a perspective view showing an individually addressableelectrode for use with the inventive HHMI having conductive stems. FIG.52 is a perspective view showing conductive stems of the individuallyaddressable electrode for use with the inventive HHMI. FIG. 53 is aperspective view showing an individually addressable electrode for usewith the inventive HHMI having the plurality of conductive elements asconductive hemispheres. FIG. 54 is a cross section showing conductivehemispheres of the individually addressable electrode for use with theinventive HHMI. The electrodes 14 may be formed, for example, byinjection molding, stamped, cast or vacuum drawn to form an elastomer,metal or plastic textured substrate. If non-conductive, the substratecan be plated or coated with a conductive material. Various electrodeshapes are possible that enhance the face-to-face electrical connectionto the user's skin, even if hair is present. For example, the pluralityof conductive elements are configured and dimensioned to provideeffective face-to-face electrical contact between the individuallyaddressable electrode and the skin of the user 12. The variousgeometries and dimensions shown herein are examples of the shapes andsizes for individual electrodes 14 that can be readily formed frommaterials and processes that enable mass production of a thin,lightweight, flexible wearable electronic that can be disposedcomfortably in direct contact with large surface areas of the user'sskin.

As shown, for example, in FIG. 24, the HHMI includes electrodes 14 incontact with the skin surface of the operator, connected via electricalleads, such as an x-y grid of conductive leads (although the x-yarrangement of conductive leads is just an example), and addressed usingtechniques borrowed from passive or active matrix video displays (at alower resolution), and various other known electronic circuits. Theelectrodes 14 are addressed so as to become an active electrical elementfor applying the electrical signals or for detecting the electricalsignals as described herein so that the HHMI can provide preciseelectrical activity detection (to detect the muscles 18 and nervesemployed in even subtle arm movement) and electrical signal application(to cause involuntary and accurate arm movement).

FIG. 55 shows an example where a specific muscle (bicep) is targeted forcontraction by applying a transcutaneous electrical signal. Theelectrical signal is applied as a DC voltage between a first electrodegroup and a second electrode group. As shown in other circuits thecircuit can be modified to include circuit elements so that theappropriate electrode group to invoke a desired muscle response can bedetermined for example, during a calibration mode. During a calibrationmode (described in more detail below), these same first electrode groupand second electrode group are used to detect the electrical activitygenerated when the user 12 performs a known action, such as raising thehand to the chest (contracting the bicep muscle). Additionally oralternatively, the appropriate electrode groups to invoke a desiredmuscle response can be extrapolated from the calibration data becausethe general physiology of a human arm 16 is well known. In this case,the calibration mode and/or refinement mode provides fine tuning of apredetermined electrode pattern, where the predetermined electrodepattern is based on human physiology and the fine tuning is based on theparticular electrical activity detected for the user 12 during thecalibration mode. A strain gauge wire or the like can also be used todetect muscle movement and/or a memory metal used to contract and applya squeezing force, acting as conductive pathways to the electrodes 14 orprovided as separate components.

The inventive haptic interface uses sensory feedback and algorithms tocorrectly control the characteristics of the electrical stimulation.Muscle contractions can be induced that result in the same movements ofthe body part of the user 12 (e.g., fingers) as if performed voluntarilyby the user 12 using precisely controlled muscle movements.

Muscle contractions and changes in body part positions can be used asmetrics during calibration and also to obtain feedback while the appliedelectrical stimulation causes an automatic and involuntary movement ofthe user's body parts. The sleeve may include transducers used tomeasure changes in muscle shape or body part position, or to apply apressure, such as a squeeze or vibration. For example, a shape memoryalloy (which could be formed as a sheath around or otherwise incommunication with the electrode lead conductors) or piezo-electric ormechanical vibrators, can be used under control of electrical signalsfrom the computer or microprocessor, to apply haptic cues in the form ofpressure or vibration.

The HHMI is constructed of layers of thin flexible materials, such asconductive stretchable fabrics, flexible insulators, flexible circuitboards, and the like. The materials may be woven, spun, closed cell,open cell, thin film, or other suitable structure.

Layers, bonded layers, and constituent elements of the HHMI may beprinted using a 3D printer, or formed by a batch or roll-to-rollmanufacturing process including lamination, screen printing, ink jetprinting, self-assembly, vapor deposited, sprayed.

The HHMI can be fabricated as a sleeve, glove, legging, shirt, full bodysuit, etc., and has a flexible and comfortable snug fit that urges theelectrodes 14 into face-to-face surface contact with the skin of theuser 12. Gel electrodes 14 can be used, but have some drawbacks. Dryelectrodes 14 are typically made from rigid materials. The electrodeconstruction described herein provides thin, flexible structuresdesigned specifically for compression face-to-face contact. Whatever thecase, the transference of the electrical signal between the electricallyconductive surface of the electrode and the skin of the user 12 has tobe effectively accommodated.

Using a drone pilot or remote operator as an example (there is moredetail on this exemplary use described herein), as shown, in FIG. 25,since every human body is different, at the beginning (step one) of acalibration mode a user 12 is asked to perform a first calibrationmotion (step two). The user 12 performs a known motion (step three) thatcauses nerve firings and muscle contractions, such as a motion thatreplicates using the hand as a control surface, e.g., a flap orthruster. In this case, the known motion can be a hand motion forming aflat plane with the fingers and bending at the wrist as if deflectingair. The electrical activity of the first motion is detected (step four)and the characteristics of the body-generated electrical activity (e.g.,electromyographic signals generated by the nerves and muscles 18 as thehand is formed into a plane and bent at the wrist) are sensed and stored(step five). In addition to the body-generated electrical activity,other physiological changes can be detected, such as a change in theshape of the user's arm 16 caused by muscle contractions. Thesephysiological changes are useful for calibrating the inventivehuman/machine interface and also for determining pilot's intendedelectrical signals. The electrical and muscle activity that is detectedand used for calibration, control intentions, user 12 conditions, etc.,can include EKG, EMG and EEG, as non-limiting examples.

A next calibration motion is indicated to the user 12 (step six), theuser 12 performs the motion (step seven) the electrical activity isdetected (step eight) and the characteristics of the detected electricalactivity is stored (step nine). If the calibration routine is notcomplete (step nine), then another next calibration motion is indicated(flow goes back to step six). If the calibration routine is complete(step nine) than a mapping is made of the detected electrical activitycharacteristics for the calibration motions (step eleven). By thisprocess, the electrical signals and the source of the electrical signals(muscles 18 and nerves) associated with known motions are calibrated forthe individual user 12 and a map of the signal characteristicsassociated with corresponding muscles 18 and nerves for each respectivecalibration motion is stored for the user 12.

In an auto-action mode, the calibration data is used to determine thecharacteristics of the computer-generated electrical activity causing adesired automatic and involuntary movement of the pilot's body parts.The result is the pilot perceives the involuntary movement as thoughcaused by an externally applied force. For example, the pilot's hand canbe calibrated as a control surface to remotely control the flaps of adrone, and the perceived externally applied force can be felt by thepilot as though the pilot's hand is the in-flight control surfacedeflecting air.

FIG. 57 is a flowchart showing an algorithm for refinement of thecalibrated HHMI using measured movement of the user 12. This exemplaryalgorithm provides for further customizing the HHMI to interface with aparticular user 12. Since every human body is different, in thecalibration mode the operator performs a known task that causes nervefirings and muscle contractions, such as a motion that replicates usingthe hand as a control surface, e.g., a flap or thruster. In this case,the known task can be a hand motion forming a flat plane with thefingers and bending at the wrist as if deflecting air. Thecharacteristics of the body-generated electrical activity (e.g.,electromyographic signals generated by the nerves and muscles 18 as thehand is formed into a plane and bent at the wrist, or when the user'sleg pushes down on a bicycle pedal) are sensed by the sensorytransducers (i.e., the electrodes 14 shown herein throughout or shown,for example, in block diagram FIG. 65). The sensory transducers are usedto calibrate the location, relative strength, etc. of each detectedelectrical signal. To refine the calibrated HHMI, a refinement processmay be started (step one). A start position of a body part, such as thehand of the user 12, is determined using for example, a known positiontaken consciously by the user 12, or the detection of the body part,such as a hand, using gyroscopes, accelerometers, IR detectors (e.g.,Leap Motion discussed herein), or others (step two). The electricalactivity resulting in the change in body part position is detected (stepthree) as the body part moves from the start position to a determinedend position (step four). For example, the hand of user 12 can bevoluntarily brought from a position where the arm 16 is relaxed and thehand is dropped down to where the hand is brought to touch the shoulderof the user 12. This motion is easily made consistent by the user 12 andallows for the determination of the start position (step two) with thehand dropped down at the user 12 side, the detection of electricalactivity that results in the change in body part position (step three)and the determination of the end position (step four) when the handtouches the shoulder. The detected electrical activity is then comparedto a stored map of electrical activity obtained, for example, using thecalibration algorithm shown in FIG. 25. The detected electrical activityand the stored map are compared to predicted the expected change inposition. The stored map is then confirmed or adjusted if necessarydepending on the comparison (step six). If the refinement is complete(step seven), the algorithm ends (step eight). If it is not complete,the refinement continues again at step two.

FIG. 58 shows the muscles 18 of a hand of the user 12. The human handincludes a complex arrangement of muscles, bones, nerves, tendons, andskin. There are more than 30 individual muscles 18 in the hand andforearm 16 that have to work together to obtain highly precisemovements. The hand muscles 18 provide great flexibility, very precisecontrol, and gripping strength allowing humans to type, write, playmusical instruments, grip a tennis racket, throw a ball, etc. FIG. 59shows the inventive HHMI configured as a pair of gloves. The fingertipsare among the most sensitive parts of the human body with thousands ofdensely packed nerve endings producing complex patterns of nervousimpulses to convey information about the size, shape and texture ofobjects. The nerve endings in the fingertips contain different types oftouch receptor organs, including Meissner corpuscles and Merkel discs.

As shown, for example, in FIG. 72 and described below, a higher densityof smaller electrodes 14 can be provided at the nerve-rich tips of theHHMI gloves. Electrical stimulation can be applied specifically to causea reaction to the fingertip receptors, so that a variety of sensationscan be perceived by the user 12 including textures, pressures, heat,cold, movement, etc.

FIG. 60 shows the mapping of individually addressable electrodes 14 tothe muscles 18 of the hand of the user 12. By targeting the respectivemuscles 18 of the hand that work with corresponding muscles 18 of theforearm 16, very subtle involuntary movements can be achieved. In thecase of tremor mitigation, these movements may act to further steady atrembling hand. In the case of an accelerated learning application,these movements may bring the fingers into position, or at least urgeand guide the fingers towards the correct position, to facilitate thelearning of a desired chord played on the keys 34 of a piano.

Specific Use Examples:

The uses of exemplary embodiments of the HHMI will be described hereinwith reference to specific applications. However, it is to be understoodthat these specific applications are to provide examples of many uses ofthe HHMI that includes the detection of electrical activity, movementand position of a body part, and/or the involuntary movement or guidanceinto position, virtual sensation, proprioception through an appliedelectrical signal.

The haptic human/machine (HHMI) is applicable to a wide range oftechniques and applications, including, military, civilian, acceleratedlearning, medical, entertainment, sporting, gaming, home automation,space and deep sea probes, as well as the remote controlled drone androbotic operation. The HHMI can also provide an immersive way tocommunicate between two people remotely located from each other, or toexperience an activity being performed or observed by another, in realtime and from previously detected and recorded data.

Military Undergarment:

FIG. 61 shows the inventive HHMI configured as an undergarment andhaving clusters of smaller, more densely packed electrodes 14 at thesolar plexus and clusters of larger, less densely packed electrodes 14located elsewhere. The solar plexus is a complex of ganglia andradiating nerves of the sympathetic nervous system at the pit of thestomach, and core functions of the body can be detected by monitoringthese structures and/or the muscles 18 in this region such as thediaphragm. The sympathetic nervous system's primary process is tostimulate the body's fight-or-flight response. In accordance with amilitary use, the HHMI undergarment can be worn by a soldier for addinga new layer of perception during, for example, a combat situation.Typically, the visual and auditory senses of a soldier are saturatedduring the high intensity of a combat situation. The HHMI undergarmentcan add a new way to convey information to the soldier using tactileinformation that can be a supplement to the audio and visual informationbeing received. The tactile information may be, for example, anindication of the location of a rallying point or where the soldier'scomrades are located. Sensors and transmitters or other data links canbe used as well to convey details about the soldier's physical conditionincluding heart rate, blood pressure, body temperature and other vitalsigns and health related conditions.

The HHMI is made from a multilayered, flexible and light weightstructure. The layers of the HHMI include compression layers that biasinward when formed in a shape that wraps around an object, such as anarm 16 when configured as a sleeve, or the user 12 back, shoulders,stomach and torso when configured as a shirt. The HHMI is a wearableelectronic with the individually addressable electrodes 14 urged intoeffective face-to-face electrical contact with the skin of the user 12.

The HHMI is a light weight, wireless, high resolution electrical signalsensing/applying wearable electronic for the detection of the usercontrol intentions (for example, to control the robot flight) and forthe application of enhanced haptic cues (for example, to experience therobot's flight conditions). The interface is in the form of acomfortable, easily worn garment that the operator wears with little orno restriction of movement.

The HHMI is constructed as a conformable, comfortable, but fairly tightfitting garment to hold the electrodes 14 in direct face-to-faceelectrical contact with the skin. The HHMI is used to apply electricalstimulation through the skin to provide haptic cues.

For example, the haptic cues may be a desired body position related to asensed parameter, such as flex, rotation, tilt, pitch, yaw, temperature,vibration, and other detectable stresses or conditions of a mechanicalcomponent (wing, fuselage, control surfaces, etc.) of the UVS.

The sensed parameter could be air pressure experienced at a wing controlsurface while maneuvering. The sensed parameter is transmitted from thedrone (using RF or line-of-sight optical), causing a computer-controlledNMES cue (electrical stimulation) resulting in an auto-action responsein the hand of the pilot feeling pressure to assume a position directlyrelated to the drone's control surface.

The pressure to move the hand is the result of muscle movements causedby the NMES cue. The pilot experiences the sensation of resistance orpressure because of the computer controlled electrical signals appliedto the pilot's own sensory/muscular physiology. In addition to pressureand resistance, the physical sensation of vibrations, knocks and evenscratches can be perceived as the result of subcutaneous controlledelectrical signal stimulation.

The muscle movements are involuntarily and automatic. There may not beany mechanical force simulators involved in creating the involuntarymovements and perceived sensations caused by the applied electricalsignals, although there can be, and mechanical force simulators mayprovided in addition to the applied electrical signals.

Mitigation of Tremor and Other Movement Disorders:

The HHMI, for example, can be configured as a wearable electronic fornon-pharmacological, non-invasive therapeutic medical use to treatdisease or illness, such as Parkinson's Disease (Parkinson's disease),essential tremor, and other neurological ailments.

In accordance with this exemplary aspect of the invention, the HHMI isconfigured to mitigate the physical and emotional difficulties of apatient suffering from a movement disorder, exemplified by, but notlimited to, Parkinsonian tremor. Parkinsonian tremor is typicallyasymmetric, occurs at rest, and becomes less prominent with voluntarymovement. The inventive HHMI offers a mechanism to conveniently apply afeedback-regulated, computer controlled, electrical signal only whenneeded to automatically counter the changing characteristics of aParkinsonian tremor.

As an example, to counteract an involuntary tremor, a microprocessor maybe programmed to detect and respond to variations in tremor onset,duration and characteristics. The exemplary circuit schematics, blockdiagrams and flowcharts shown herein illustrate possible components ofthe inventive system that includes the HHMI and its constituent parts.Additional circuitry and components may be employed, as necessary, toachieve a specific utilization. For example, the signal generation mayoccur though the microprocessor and related circuitry 7 or themicroprocessor may control an external signal generators capable ofcreating an appropriate mono or bi-phasic electrical signal that iseffective to cause at least an urging towards a predetermined motionand/or a predetermined position of a body part a known in the art. Anexample of such a signal generator, known as a TENS unit, iscommercially available from companies such as Amrex, BioLedical LifeSystems, Prizm Medical, and others.

An exemplary embodiment of the HHMI is constructed as a thin, flexiblesleeve unobtrusively worn by the user 12, and the connection between thesleeve and microprocessor can be direct or via wireless networking, suchas optical, or RF (e.g., Bluetooth, WiFi, etc.). The HHMI may beembodied in a lightweight, comfortable, haptic sleeve having electrodesize and density enabling automatic calibration to the unique physiologyof a user 12.

The haptic sleeve provides precise electrical activity detection todetect the muscles 18 and nerves involved in, for example, arm movement.Additional detection of body part movement can be made using IRreflectors, emitters incorporated in the HHMI or worn on the body part,and a CCD or other detector located remotely from the moving body part.Accelerometers, proximity sensors (Hall effect, sonar, etc.), gyroscopesand other motion and location sensors can also be used to detect bodypart movement. For example, in an accelerated learning example, usingthe HHMI, subtle hand movements of a pianist can be determined by theHHMI detecting electrical activity of the pianist and computerprocessing of a video signal of the pianist's hands while playing thepiano. These two sources of data can be used to determine the locationsof the instrument relative to the performer, as well as to obtain a goodindication of the fingers and hand movements so that the appropriateelectrical signals can be determined and applied, via the HHMI to astudent. Motion controllers, such as commercially available productsfrom companies such as Leap Motion of San Fransisco, Calif., can be usedto determine, without requiring the wearing of any reflectors oremitters, the finger and hand position of the pianist nearlyinstantaneously. Similar systems can be used for detecting the movement,location and positions of other body parts as necessary to effect thevarious embodiments described herein. A MIDI keyboard can be used toprovide data on the actual keys 34 struck by the fingers of the pianist.

In this medical use example, rhythmic motions can be detected, thecharacteristics stored and analyzed, so that the slight movementsindicating the onset and oscillation of a tremor or other undesired (ordesired) body part movement can be determined. In an exemplary medicaluse case, in response to the detected tremor, the HHMI sleeve applies anearly instantaneous electrical signal to cause selective involuntarymuscle and nerve impulses that counteract and negate the undesirablearm/hand trembling that would have otherwise occurred.

Because the applied signal oscillates in response to the tremor, theapplied electrical signal results in muscle contraction perceived as arhythmic, massage sensation by the user 12. Among the advantages of thisapproach is immediate and automatic relief without drugs or surgery. Thetremor is only counteracted when the patient wears the wearableelectronic device.

Usually, one of the most noticeable outward signs of Parkinson's diseaseis involuntary movement of the limbs. The disability caused by tremorcan be anything from embarrassment to a total lack of independence. Thedrugs used to treat tremor are numerous, (e.g., xanax, topamax,neurontin, propranolol, topiramate, primidone, mysoline) and most haveundesired side effects, which may not be tolerated by a particularpatient. The surgical solution is the implantation of a deep brainstimulator in a highly invasive procedure, which is usually onlyresorted to if the tremor is extremely severe.

An exemplary embodiment of the HHMI can be used in conjunction withtraditional therapeutic choices, and may reduce the dosage oraggressiveness of the drugs conventionally used to treat tremor, and mayobviate or at least delay the need for invasive deep brain surgery. Thecontractions and flexing caused by the applied electrical signal mayalso provide relief to stiff and aching muscles 18 which usuallyaccompany Parkinson's disease illness, and/or the HHMI can have aspecific “stretching and massage” mode where the applied electricalsignal is intended to create a rhythmic massaging action by contractingand relaxing the user 12 muscles 18.

In addition to the mitigation of tremor, the HHMI may also be useful toprovide drug-free, non-invasive relief to gait disturbance anddisequilibrium. In this case, the HHMI can be scaled up and worn as alegging or body suit that steadies the patient from involuntarymovements of the legs or other body parts.

In the case of tremor mitigation, the target of the HHMI is the muscles18 and nerves involved in involuntary oscillatory motion. The HHMI canbe utilize for electromyography as a simple and quick method tocharacterize muscle and nerve electrical signals, such as those causedby voluntary movements like playing a musical instrument, or involuntarymovements like tremor. For example, the HHMI can be used in calculatingtremor frequency and amplitude for assisting diagnosis. The HHMI can beused for detecting movement from electrical activity generated by themuscles 18 and nerves, and applying electrical stimulation to guidefingers into the position of a musical chord, to counteract tremormovement.

The human body can be considered a highly complex but modular machine,with muscles 18 that react consistently and predictably to an appliedelectrical signal. Although humans all share the same general anatomy,the specific locations for optimal electrode placement and the appliedsignal characteristics will vary significantly from person to person,and even from use to use by the same person. In an exemplary embodimentof a high Definition HHMI, the number of addressable electrodes 14 isincreased, and can be as small as square millimeter or less although itis expected that square centimeter or larger electrodes 14 will beadequate for most muscles 18 and body parts such as the forearm 16,while smaller electrodes 14 may be better used for example, to detectand apply electrical signals to the muscles in the hand and the nerveslocated at the finger tips.

Also, higher density, smaller electrode clusters may be disposed at thelocations of muscles 18 and nerves consistent with general humananatomy. The HHMI can have a mix of electrode sizes and shapes andneeded, for example, to apply desired neuromuscular electricalstimulation.

The use of transcutaneous electrical nerve stimulation has been approvedby the FDA for pain relief, and more recently for preventing migraineheadaches. Others have offered some general mechanisms for usingelectrical stimulation to combat tremor. However, none disclose orsuggest a wearable electronic device for precisely detecting electricalmuscle and nerve activity using very small, individually addressedelectrodes 14, and applying complex electrical signals via those same orother electrodes 14 to mitigate Parkinsonian tremor as the HHMI. Nor doany of the prior attempts indicate the use of a haptic human/machineinterface to provide haptic cues that are synchronized withcomputer-controlled audio and video cues to strengthen the brain'swiring of the voluntary motor control that is being taken over, forexample, by the involuntary Parkinsonian tremor.

U.S. Pat. No. 6,892,098 B2, entitled Nerve Stimulation for TreatingSpasticity, Tremor, Muscle Weakness, and Other Motor Disorders, issuedto Shai, et al. discloses a method for treating spasticity by driving acurrent into a nerve to inhibit propagation of action potentials in oneor more sensory fiber. U.S. Pat. No. 7,228,178 B2, entitled Surfacestimulation for tremor control, issued to Carroll, et al. disclosesnon-invasive electrical stimulation of the brain through skin surfacestimulation of the peripheral nervous system as a treatment for movementdisorders. US Pat. App. No. 20030149457 A1, entitled ResponsiveElectrical Stimulation for Movement Disorders, issued to Tcheng, et al.discloses an implantable neurostimulator system for treating movementdisorders. PCT Pat. App. No. PCT/US2014/012388, entitled Devices andMethod for Controlling Tremor, issued to Rosenbluth et al. discloses theelectrical stimulation of a peripheral nerve to combat tremor.

The HHMI provides a means for mitigating the embarrassing and oftenseverely debilitating outward effects suffered by a Parkinson's diseasepatient. But this is just the beginning for the potential use of theHHMI to combat the psychological and physiological degeneration causedby Parkinson's disease. In accordance with other aspects of the HHMI, atherapeutic treatment addresses the cognitive impairment caused byParkinson's disease and other neurological disorders. The HHMI provideshaptic cues synchronized with computer-controlled audio and video cuesto strengthen the brain's wiring of the voluntary motor control that istaken over by the involuntary Parkinsonian tremor.

The haptic sensory cues may stimulate a processing center of the user 12brain to form a learned response to the involuntary tremor, the learnedresponse being reinforceable over time to mitigate the involuntarytremor.

The plasticity of the human brain is only now being realized. Thistherapeutic use of the HHMI may strengthen the neurological pathways inaddition to re-enforcing the patient's ability to combat resting tremor.This rewiring of the patient's brain may be effective in furthercombating cognitive problems including dementia and thinkingdifficulties; and emotional changes, such as depression, fear, anxietyand loss of motivation. In accordance with this aspect of the invention,the sensory cues can be utilized to provide rehabilitation to a victimof a brain injury or other brain damage or learning dysfunction.

Accelerated Learning of a Musical Instrument

As shown, for example, in FIG. 62, the HHMI can be used to indirectly ordirectly transfer the nuances of a performer's musical skills andpassion to students, and to multitudes of people, young and old,throughout the world and down through the generations. The HHMI may beused as a component in an Accelerated Learning System (ALS) that usescomputer-controlled sensory stimulation that is synchronized andreceived by multiple senses of the student to more quickly build themuscle memory and pattern recognition necessary to learn an instrument.For example, audio cues (a piano melody) are combined with visual cues(image of a performer's fingers and hands correctly playing the pianomelody) and haptic cues (vibration and/or electro-stimulation of themuscles/nerves of the student's fingers corresponding to the relevantmuscles/nerves of the performer). This ALS stimulates the separatesensory processing centers of the brain to re-enforce and hardwire thebrain/nerves/muscles 18 needed to learn and master the instrument, andthe learning session can be done at any time, at the instrument or away,even while engaged in another activity.

Haptic sensory cues applied via the HHMI are dependent on hand positiondata and finger strike data related to the hand and finger position offingers that strike one or more keys 34 of a musical instrument. Thehaptic sensory cues can include an electrical signal effective to causeinvoluntary body part motion and cause a perceived somatic and/orkinesthetic sensation so that the hand of the user 12 is urged towards ahand position to form an intended chord and the fingers that strike thekeys 34 to make the chord receive a touch sensation, whereby a musclememory may be created in the user 12 reinforcing the hand position andkey strikes.

The specific nuances and personal stylistic details of a performance arecaptured and at any later time provided as synchronized sensory cues toa student (or even an entire audience) based on this recordedinformation. The HHMI can capture the subtleties that are the differencebetween a musician and a virtuoso, down to the timing and coordinatemuscle movements that result in an artist's unique style.

In accordance with another aspect of the invention, a plurality ofhaptic sensory cues are generated capable of being perceived by a user12. The plurality of haptic sensory cues are dependent on a determinedcondition of at least one movable member of a performing body performingan event. The plurality of haptic sensory cues are effective forstimulating a touch processing center of a brain of the user 12 based onthe determined condition. A plurality of visual sensory cues aregenerated capable of being displayed to the user 12 on a video displaydevice. The visual sensory cues provide a virtual visual indication tothe user 12 of a position of at least one of the at least one moveablemember and the performing body. The visual sensory cues are effectivefor stimulating the visual processing center of the brain of the user12. The visual sensory cues may be synchronized with the haptic sensorycues so that the position is virtually visually indicated insynchronization with the haptic sensory cues and so that the visualprocessing center is stimulated with the visual sensory cues insynchronization with the haptic sensory cues stimulating the touchprocessing center.

The synchronized stimulation of the touch processing center and thevisual processing center can be used for teaching the user 12 to performa version of the event. The synchronized stimulation of the touchprocessing center and the visual processing center can be used forenabling the user 12 to remotely control the performing body performingthe event. The performing body may be a human, and the movable membermay comprise a body part of the human. The performing body may comprisean animal, and the moveable member may comprise a body part of theanimal.

The performing body may be a remotely controlled moving object, such asa drone or robot, and the moveable member may comprise a mechanicalcomponent of the remotely controlled moving object, such as a controlsurface or thruster. As shown in FIG. 62, the movable member may be afinger, the performing body may be a human, and the event may be playinga piece of music. The movable member may be a control surface, theperforming body may be a drone, and the event may be flying the drone.

FIG. 62 shows an embodiment of a tactile information and visualinformation system used to learn to play the piano. In this case, theuser 12 may wear a glove the produces a tactile stimulation including atleast one of an electrical signal applied to illicit a virtual somaticsystem sensation or a physical sensation generator such as vibrators.When, for example, learning to play the piano, tactile stimulation,along with a view of the real world scene, along with a superimposedaugmented reality of a virtual visual indication (such as a video of ahand with fingers pressing corresponding keys 34) can be used eitherwhen sitting at the piano and learning to play the piece of music orremotely from the instrument to construct the associative memoryfacilitating in learning the piece of music.

For example, the memories associated with the playing of a piece ofmusic, in accordance with an embodiment of the invention, will bere-enforced by audio, visual, and haptic (muscle, tactile or otherstimulation) cues that are generated and that can be repeated over andover to instill the associative memory that is built up during thecourse of conventional music practice at an instrument. Thus, the user12 can virtually learn to play the instrument, and the distinct memorycues of vision, hearing, and touch will build up muscle memory in“automatic” type of memory installments.

In accordance with an exemplary embodiment, a user 12 (performer and/orstudent) wears haptic/visual gloves that indicate to the user 12 thehaptic and visual cues corresponding to the learning session, forexample, the playing of a piece of music during the recording andlearning of the learning session. The user 12 may also wear headphonesto hear the audio cues (or, the audio cues may be presented fromspeakers or other auditory stimulating mechanism). The user 12 mayfurther wear a pair of visual/audio recording/display glasses orgoggles, available, for example, from/as Google glass, Oculus Rift,Microsoft Hololens, and Meta VR goggles.

Thus, as will be described in more detail below, the user 12 receivesdifferent audio, visual, and haptic cues to indicate the notes or keys34 being played, for example, on a piano during a learned piece ofmusic. For example, the user 12 may receive visual cues through thelighting up of LED lights on the fingertips that correspond to thefingers playing a key on a piano. Simultaneously, or alternatively, thenerves of the skin and muscles 18 corresponding to the finger may bestimulated via vibrations or electrical impulses so that muscle memoryof the learned piece of music is built up in conjunction with theauditory and visual cues.

Using the inventive ALS, a student receives simultaneous sensory cues,which may be similar to the sensory cues that are received during anactual practice session at an instrument, or that may be different thanan actual practice session experience. As an example of an actualpractice session experience, a visual sensory cue may be a displayshowing the keyboard with the hand positions from the perspective of theperformer. FIG. 62 illustrates for example a visual perspective of thekeyboard that can be displayed to the student as a visual sensory cue,in this case the visual perspective corresponds to the viewpoint of theperformer. In this case, the previous recording of the visual cues canbe done using a head-mounted video camera 38, or a pair of augmentedreality glasses that include a video camera 38. As an example of asensory cue that is different than an actual practice sessionexperience, a visual sensory cue can be artificially generated thatcorresponds to the learning session. In this case, for example, images,(such as the keyboard pattern when a chord is correctly played, or themusic notation indicating which keys 34 to play that make up the chord)can be display to the student in synchronization with the audio sensorycues and/or the haptic sensory cues applied during the learning session.Further, different versions of the same type of sensory cues can beapplied simultaneously. In this case, as an example, the viewpoint ofthe hand of the performer can be displayed in the center of thestudent's field of view while at the same time the generated sequence ofimages can be displayed in a peripheral portion of the student's fieldof view. Head tracking can be used to further immerse the student in theremotality session, allowing the student to naturally shift focus onvirtual and/or actual keyboard, hands and sheet music. Simultaneously,the corresponding audio sensory cues and haptic sensory cues can beprovided to the student.

Further, the audio sensory cues and the haptic sensory cues may also bevaried to achieve a specific goal. For example, the left hemispherecontrols movement on the right side of the body, so the audio sensorycues corresponding to the haptic sensory cues applied to right hand maybe applied to the left ear of the student, or vice-versa, depending onthe portions of the brain that are desired to be simultaneouslystimulated. These generated sensory cues will be received by differentparts of the user 12 s brain, to create an associated processing andmemories between the various parts of the brain that are stimulated. Thestudent, or person being rehabilitated, or person being entertained,experiences, for example, the piece of music with the reinforcement ofthe associated memories resulting from the simultaneously appliedsensory cues. This experience can occur during practice sessions at aninstrument and/or remotely from the instrument.

In accordance with an embodiment of the inventive accelerated learningsystem, to further enhance the learning experience, chemicals releasedor electrical signals generated by the brain systems can be detectedfrom a student that is actually learning the piece of music at apractice session at the instrument. As another example, the brainactivity of a student can be sensed using well-known brain scantechniques (such as event related potential, ERP) and the appliedsensory cues can be the focus of the different brain activities relatedto auditory, visual, and haptic sensory cue processing to furtherreinforce and enhance the learning experience. The inventive ALS can beapplied to other activities, including but not limited to sports, schoolwork, performing arts, military exercises, video gaming, etc.

An embodiment described herein pertains to learning to play music on akeyboard. However, the inventive accelerated learning system is not atall limited to keyboards, or to learning to play music. As is alsodescribed herein, aspects of the inventive accelerated learning systemcan be utilized for a number of different fields, includingentertainment, military, sports, video gaming, remote controlled robots,drones and vehicles, other musical instruments, etc.

The HHMI utilized as a musical teaching aid can be part of an effectivetherapy for stroke injury and other brain damage that can berehabilitated through rewiring of the damaged brain using thesynchronized, computer-controlled haptic, audio and visual cues.

To illustrate the basic HHMI signal detection and applicationcomponents, FIG. 63 shows a simple signal detection and applicationwhere an EMG sensor is located at the belly of the extensor digitorum. ATENS signal-applying electrode is located at either end of the muscle. Amovement detecting accelerometer is located on the back of the hand. Inthe actual HHMI system, the sensors and electrodes are more numerous,with an optimal size, number, type and shape of the electrodes dependenton the particular application.

Sports Training

FIG. 64 shows data collection on a bicycle for use in sports training. Amicroprocessor controls the reception of data from sources such as aGPS, microphone 36, camera 38, gyroscopes, accelerometers, speedometers,etc. This data is logged by a data logger and stored in a memory. FIG.65 shows the synchronized application of sensory cues dependent on thedata collection of FIG. 64 during a training session. During thetraining session, the logged data is retrieved from memory and used bythe microprocessor to determine a plurality of first sensory cues. Thesensory cues are generated by the microprocessor and made available forperception by the user 12, for example, the sensory cues can be aspherical point of view video that is viewable from all angles usinghead tracking AR goggles. The plurality of first sensory cue aretime-sequentially generated and effective for stimulating at least onesense of the user 12, such as vision. A plurality of haptic sensory cuesare generated capable of being perceived by the user 12. The hapticsensory cues may be received by the user 12 dependent on computercontrolled time-sequentially generated electrical signals. For example,the haptic sensory cues can be applied using the HHMI and create thesensation of wind rushing over the arm 16 s of the user 12 where thewind speed is dependent on the virtual cycling speed. Additionally, oralternatively, the haptic sensory cues can be a varying resistance forceapplied to resist the pedaling motion of the user 12 as if the user 12is cycling up and down the hills of the course. These electrical signalsinvoke a perception by the user 12 related to the sense of touch. Thehaptic sensory cues are generated in synchronization dependent on thetime-sequentially generated plurality of first sensory cues. Forexample, as the speed of the bicycle goes faster, as indicated to theuser 12 by the scene on the VR goggles, the apparent wind speed on thearm 16 s of the user 12 also increases by applying an appropriatecomputer controlled haptic signal.

FIG. 66 shows the collection of data sampled along a route taken by acyclist. FIG. 67 is an isolated view of the collection of data sampledalong the route showing the bicycle at an angle and height relative tosea level. The collected data is used by the microprocessor, forexample, shown in FIG. 65, to calculate a resistance value to be appliedby a resistance controller to a training system of a stationary bicycle.FIG. 68 is graph showing the collection of data as exemplary altitudeand angle relative to sea level data collected over time along the routetaken by the cyclist. The resistance may be, for example, calculateddependent on data such as the angle of the bicycle relative tohorizontal (e.g., the steepness of the hill when the actual cycle routeis taken), user weight, GPS data, speed, etc. Some of the data can bedirectly collected during the ride along the route orcalculated/approximated by the microprocessor. As an example, a cyclistcan train on a stationary bicycle but experience the legs of the Tour deFrance as a virtual experience.

FIG. 70 illustrates an augmented reality visual sensory cue showing anactual tennis racket seen from the perspective of the user 12 with anvideo overlay of a virtual tennis ball generated using computer programcode and displayed using an augmented reality display, such as augmentedreality goggles. A performance object, in this case, a tennis racket andthe position of the performance object can be detected by appropriateproximity sensor, motion detectors, tilt detectors, a laser positioningsystem, and other mechanisms used to detect the position of an object inthree-dimensional space. A performance element in this case may be thehandle of the tennis racket, and its position relative to an arm of theuser as a tennis ball approaches and is struck by the racket can bedetermined. The tennis ball can be an actual tennis ball (in the case ofaugmented reality), or a computer generated tennis ball (in the case ofvirtual reality) that the user sees and reacts to during a recording ofthe sensory cues that will be used to teach the performance. Thisexemplary mechanism and method for detecting and recording the positionof body parts and performance objects/performance elements is used torecord the sensory cues that are used to teach the event and build upmemory associations of the event in the various processing centers ofthe human brain. The body member that is detected during the recordingof the event performance (for example, the body part of a tennis pro)and then stimulated during the learning lesson or entertainment session(for example, the body part of a tennis student) can be at least one ofa finger, toe, hand, foot arm, leg, shoulder, head, ears and eyes. Thistechnique of using the inventive accelerated learning system can beused, for example, to create a virtual sport video game. Similaralternatives can be constructed for other events, such as controlling aremotely controllable system, for example, the flying of a droneairship, a space exploration probe, the playing of a guitar, theassembly of a weapon, entertainment or brain rehabilitation to help“rewire” the brain of a stroke victim or brain damaged patient, othercognitive therapy including enhanced learning, or any other event wherea user can benefit from recorded sensory cues that stimulate the variousprocessing centers of the brain.

Entertainment

FIG. 69 illustrates a chair configured for an exemplary entertainmentapplication. The haptic sensory cues can be mapped to a chair, bed,clothing or apparatus that can be worn by the user 12. A “massage”chair, for example, may have zones corresponding to areas of the bodythat can be individually applied with haptic data such as vibration orpressure. For example, in the case of a massage chair, a soothingmassage can be applied where vibration and pressure applied to variousparts of the body are mapped to the different frequencies of a piece ofmusic. The different ranges of music frequency can also be mapped tovisual stimulation in the form of light colors. The light colors cancorrespond, for example, to the sensitivity of the human eye to colorstimulation. Thus, for example, the color can be generated by LED lightsthat match the peak wavelength sensitivity of the cones of the humaneye. The three types of cones have peak wavelengths near 564-580 nm,534-545 nm, and 420-440 nm, respectively. Head tracking goggles,binaural headphones and the haptic chair can be used to provide a deepimmersion into a remotality that provides entertainment and learningexperiences, etc., in a comfortable manner.

FIG. 71 shows a user 12 experiencing deep immersion of a virtualreality, a block diagram showing detection and application of data, andillustrating the processing centers of the brain stimulated by theapplied synchronized sensory cues.

As shown, for example, in FIG. 71, the various portions of the brainrelated to the processing of sound, touch and vision can be controllablyand simultaneously stimulated so that a weakened brain sensory, motor orcognitive processing center can be strengthen or rewired through thesupport of stronger brain sensory stimulation processing centers. Forexample, a stroke victim with damage to right side of the brain may havea loss of function in the motor control of the fingers of the left hand.In this case, the haptic sensory cues applied to the fingers of the lefthand provide touch sensory stimulation to the damaged portions of thebrain, while the corresponding visual and audio cues reinforce there-learning or rewiring of the damaged portions of the brain through thetouch sensory stimulation.

In accordance with another aspect of the invention, a plurality of firstsensory cues are generated capable of being perceived by a user 12. Theplurality of first sensory cue are time-sequentially generated andeffective for stimulating at least one sense of the user 12. A pluralityof haptic sensory cues may be generated capable of being perceived bythe user 12. The haptic sensory cues may be received by the user 12dependent on computer controlled time-sequentially generated electricalsignals. The electrical signals invoke a perception by the user 12related to, for example, the sense of touch. The haptic sensory cues maybe generated in synchronization dependent on the time-sequentiallygenerated plurality of first sensory cues.

The first plurality of sensory cues comprise visual sensory cues forproviding a virtual visual indication to the user 12 of an event. Thevisual sensory cues may include video data mapped to at least one ofcolor and intensity of an image of the event, and wherein the hapticsensory cues are generated dependent on the mapped video data. The firstplurality of sensory cues may comprise auditory sensory cues forproviding a virtual auditory indication to the user 12 of the event. Theauditory sensory cues may include sound data mapped to stereo,multichannel and/or binaural channels; and wherein the haptic sensorycues are generated dependent on mapped sound data. A plurality ofconductive patches may be provided for applying an electrical signalthrough the skin of a user 12 to stimulate electrical signal receptors.

A signal generator may be provided for generating a plurality of hapticcues in the form of electrical signals applied to the skin of the user12 through the plurality of conductive patches, wherein the plurality ofhaptic sensory cues are capable of being perceived as a predeterminedsensation or muscle movement of the user 12. The plurality of electricalsignals may have at least one characteristic including location, timing,pulse length, frequency and amplitude effective to cause at least one ofthe predetermined sensation and muscle movement in the user 12. Theelectrical signal receptors may comprise at least one of muscles 18,nerves and touch receptors. The signal generator may further generate aplurality of first sensory cues capable of being perceived by a user 12,the plurality of first sensory cue being time-sequentially generated andeffective for stimulating at least one sense of the user 12; and whereinthe plurality of haptic cues are time-sequentially generated insynchronization dependent on the time-sequentially generated pluralityof first sensory cues.

A conductor grid may be provided for electrical communication of theplurality of electrical signals from the signal generator to theconductive patches, whereby the conductive patches are driving as amatrix of individually addressable electrodes 14.

In accordance with another aspect of the invention, an apparatuscomprises at least one processor, and at least one memory includingcomputer program code. The at least one memory and the computer programcode are configured to, with the at least one processor, cause theapparatus at least to detect a body part movement of a user 12 using ahuman/machine interface. Electrical signals are determined havingelectrical characteristics effective to cause a desired action inresponse to the detached body part movement, and the electrical signalsare applied to an object to cause the desired action. The desired actionmay be, for example, the future movement of a student's hand where thehand is the object.

The sensory cues can also include other senses, such as taste and smell.In this case, the senses of taste and/or smell can be utilized toprovide positive and negative reinforcement of a learned activity. Forexample, in the case of a drone operator learning to determine how torecognize friend or foe, during a training exercise a visual sightingthat challenges the operator with making a correct snap determination offriend or foe can be reinforced by providing a pleasant smell when acorrect determination is made and an unpleasant smell when an incorrectdetermination is made. By this application of additional sensory cues asreinforcement to learned behavior or responses, another processingcenter of the brain is brought into the combined sensory processinglearning experience.

In accordance with another aspect of the invention, a non-transitorycomputer readable memory medium stores computer program instructionswhich, when executed, perform operations described here.

Gaming

FIGS. 72-75 illustrate a use of the inventive HHMI and acceleratedlearning system (ALS) for teaching and/or improving hand-eyecoordination for a control device such as a joystick, remote controlunit and/or video game controller. In the case of a gaming controller,for example, the typical haptic feedback may be applied in addition tothe haptic sensory cues provided by the inventive accelerated learningsystem. For example, the rumble pack of a video game can be used toprovide further sensory information during the learning exercise. Inthis case, the rumble pack may be simulated by an additional vibratordisposed in the palm or on the back of the hand of the haptic gloves. Inaccordance with an embodiment of the inventive accelerated learningsystem, a drone operator can be placed, for example, into a sensorydeprivation chamber, during the learning sessions and/or during actualdrone flights. The nerve endings and muscles 18 of the user 12 can bestimulated by vibration or electrical impulse. Also, the electricalimpulses traveling on the nerves and the muscle movements in response tothose impulses can be detected to record a performance for uses such asa learning session, and/or to detect that a student is correctlyapplying the learned skill or behavior, and/or to provide cognitive andphysical therapy to a patient.

FIG. 72 shows the inventive HHMI configured as a glove having smaller,higher density, higher resolution, smaller electrodes 14 disposed atnerve-rich finger tips of the user 12. The HHMI can be composed of aglove having tactile finger tips. The tactile finger tips can beconstructed similar to the haptic sleeve and suit shown herein to map,detect and apply electrical activity at the user 12 fingers.

FIG. 73 shows the inventive HHMI configured as a sleeve and applied as aretrofit modification or OEM device in signal communication with agaming controller. The HHMI may communicate over a wireless or wiredconnection with a console or hand controller, such as an X-box,Playstation, Wii, Nintendo, or other gaming platform. The typical gamingcontroller includes a vibrating element (sometimes called a “rumblepack”). Much of the gaming software makes use of the rumble pack toprovide haptic feedback, for example, to provide a somatic vibratingsensation when a grenade explodes, or a rocket ship takes off, or a carengine revs. In accordance with this aspect of the invention, the HHMIcan make use of the control of the rumble pack during game play of anexisting game or using code written specifically for the HHMI so that ahaptic cue is applied to the user 12. A microprocessor may be used togenerate a specific haptic cue corresponding to the software code makingup the game.

FIG. 74 illustrates a virtual reality controller having haptic pads forapplying electro-tactile sensations to the finger tips of a user 12. TheHHMI may be formed as an orb 40 having haptic and pressure active fingergrooves. The orb 40 may have high resolution haptic and pressure activefinger grooves.

FIG. 75 is a block diagram illustrating an embodiment of the inventiveorb 40 for detecting data including user-applied pressure, bio-generatedelectrical signals, bio-active electrical signals, and changes inposition and accelerations of the virtual reality controller. and otherelements of a hand-controlled wireless haptic information transducer ofthe inventive human/machine interface. In accordance with thisnon-limiting exemplary embodiment, transducers are provided fordetecting and applying electrical signals to the finger tips of the user12. A hand operated orb 40 can include finger grooves that receive eachfinger and are lined with the transducers for applying and receivingelectrical energy and other tactile stimulation (e.g., vibrations orpressure). The orb 40 comprises a housing that holds transducers,accelerometers, microprocessors, vibrators, gyros, and transmitters,etc., enabling the use of the orb 40 as a human/machine interface suchas a wireless three dimensional mouse or wireless joystick-likecontroller for gaming, entertainment, military, business, remote controland many other uses.

As shown in FIG. 76, for example, the electrical stimulation can beapplied as a varied electrical signal that simulates the effects ofgravity (for example, moving the hand and arm 16 as if catching a ballor even the ball hitting the hand with a jolt), or as another example,as a low frequency, variable intensity pulse, synchronized to counter adetected tremor oscillation. The electrical signal triggers, forexample, the alpha motor nerves that cause muscle movement. The higherthe intensity of the electrical stimulus, the more muscle fibers will beexcited, resulting in a stronger induced contraction. The inducedcontractions can have different speeds and duration dependent on thecharacteristics of the applied electrical signal, for example, asnecessary to simulate the force of the moving ball or to steady thetremor.

To accurately target the muscles 18 and nerves, the location for signaldetection and signal application that is unique to an individual user 12is accommodated. The HHMI enables accurate detection of the nuances ofthe relevant electrical signals generated by the human body, uses thecomputational power of microprocessor(s) to process these detectedsignals, and determines and generates a complex responding signalwaveform applied to directly and effectively guide the body to a desiredposition, replicate the touch and movement sensations of a real-worldobject being virtually encountered by the body, or counter a detectedtremor.

The inventive orb 40 with high resolution haptic pressure active fingergrooves with solid state pressure sensors can be on/off or gradient. Therumble pack signal may be used to trigger the haptic sensation, theaudio sound track used to trigger the haptic sensation, gaming soundsand video (loud explosions or bursts of light of a certain colorindicating an explosion) can be used for detection to trigger hapticsensation, or a user-determined sound (filtering) can be used to triggerthe haptic sensation.

Human/Human Virtual Interaction

FIG. 75 illustrates the inventive HHMI with synchronized haptic, audioand video signals dependent on a virtual or augmented reality anddependent on actions of a remotely located second user 12 for creating ahuman/human interface. To simulate a sudden force, like catching a ball,the touch sensing cells of the user 12 are activated to simulate adetected force and cause an involuntary movement that is dependent on adetermined weight, trajectory, and speed of a virtual ball as it isperceived to contact the user 12 hand (accompanied with sound and visualcues of the virtual ball and the contact with the skin). To avoidmiscues, the signal from muscle movement may be determine fromelectrodes 14 that receive signal above a noise threshold.

USER 1 tosses a virtual ball, using the laws of physics the processordetermines the ball's trajectory based on its virtual weight, airresistance, gravity, etc. Visual and audio cues are determined based onthe ball's virtual appearance, the determined trajectory and othervariables such as a virtual gust of wind, and both USER 1 and USER 2 seeand hear the ball according their perspectives.

The HHMI is used to apply haptic cues including resistance to motion(inertia) experienced by USER1 and the sudden impact of the ball in thepalm of USER2 when it is caught. The HHMI is used to create anexperience for the USERs as if they are tossing the virtual ball betweeneach other, even though they may be connected through the network as fardistant physical locations.

Robotic and Drone Remote Control and Sensing:

Drones can be enabled to fly for prolonged duration and travel longdistances at high speed with an energy efficient on-board computer andflight control system, and with no communications to the drone aftertake off. Such autonomous drones may even navigate through doorways andwindows. The HHMI can be configured as an undergarment that is used withother VR or AR component to creates an electrically applied sensationtied into the in-flight drone conditions, like the change in angle(e.g., banking) haptic indicator mentioned in my last email. The pilothas a visual sphere from drone, and can view landscape from any angle.Computer extrapolation can put pilots point of view outside of drone,allowing the pilot to see the drone in conjunction with its movingposition relative to the landmarks. The computer or microprocessor usesflight condition data from on-board detectors to generate correspondinghaptic signals experienced by pilot. Haptic feedback give pilotreal-time sense of drone orientation and flight conditions, as if thepilot is the drone. For example, as an entertainment use a virtualpassenger can go “along for the ride.” Or, in a war fighter situation, adrone that is enabled with advanced autonomous acrobatic flightcapability can also include the control by a remote pilot that isrelieved of at least some of the requirements for maintaining flight,allowing the pilot freedom to focus on aggressive dog fight maneuvers.In the case of a land robot, an autonomous robot may include activecontrol of a remote operator that focuses on searching for a victimthrough a burning building.

FIG. 77 shows the human/robotic interface that uses the physiology ofthe human to integrate the onboard and ambient conditions of a remotelyoperated flying robot into the information pool used to control therobot. In addition to a view from what would be the cockpit of the drone(if it had a human occupant), an artificial real-time perspective viewof the drone may be displayed on a visual cue system (e.g., virtualreality goggles). Data from onboard cameras 38 plus onboardaccelerometers, gyroscopes, GPS, etc., plus stored image data of thedrone can be used to create real-time artificial perspective of drone inflight that is received as the visual sensory cues. The pilot canperceive the visual image of the drone as if flying alongside the drone(e.g., information with the drone). Alternatively, the drone and thescene around the drone can appear to the pilot from any other visualperspective. This same basic system can be used, for example to providehuman/human virtual interaction and human/instrument (tool, implement,etc) interfacing similar to the human/robotic interface describedherein, but where another human or physical implement (for example, forrobotic surgery) is interfaced with the user 12. Multiple sources ofdata can be used in the HHMI, including onboard real time, filtered,analyzed, prerecorded, computer generated or otherwise received,obtained or created. For example to obtain video data, a spherical viewcamera 38 system can be for collecting video information directlyonboard a remote machine, such as a drone. This video data can be storedand used later, for example, in gaming or a training exercise, or it maybe used in real or near-real time such as during a drone combat missionor robotic search and rescue operation.

In accordance with this non-limiting, exemplary embodiment the collectedtime sequential data (e.g., audio, video and haptic signals) aretransmitted from the drone to the pilot and the flight electricalsignals transmitted from the pilot to the drone.

At the location of the pilot, a plurality of haptic sensory cues aregenerated capable of being perceived by the pilot. The haptic sensorycues are received by the pilot as computer controlled serially generatedelectrical signals via the HHMI. The electrical signals invoke aperception by the pilot related to the sense of touch. These receivedhaptic sensory cues can be applied as computer controlled electricalsignals that are mapped to the body of the pilot so that different bodyparts receive different sensory stimulation. For example, the hands andarm 16 s of the pilot may be considered the human embodiment of thecontrol surfaces, such as flaps of a drone plane. The feet and legs ofthe pilot may be considered the human embodiment of propulsioncomponents, such as the engines of the drone plane. In this example, theflexing of one or both feet of the pilot can be detected and convertedto flight electrical signals to control the engine speed (and therebycontrol the speed of the drone). Engine speed time sequential datareceived from the drone can be converted into a haptic sensory cue thatis displayed along with visual speed data, such as GPS determined speedrelative to ground, so that the pilot has an intuitive sense of thedrone engine speed (for example, intensity of a sensed vibration can becorrelated with the RPM of the engine) and along with the visualconfirmation of the drone speed relative to ground. In accordance withthe inventive human/machine interface, the pilot receives multiplesensory cues that are inter-related and synchronized to indicate theflight conditions of the remote drone.

As shown, for example, in FIGS. 77-80, an exemplary use of the HHMI isfor controlling a remote machine, such as a robot or a drone. FIG. 77illustrates the inventive HHMI for remote sensing and controlling of adrone. The inventive HHMI adds a novel dimension to the human/machineinterface. For example, the HHMI utilizes haptic sensory feedbackcreating relevant touch cues related the remote robot onboard/ambientconditions. The HHMI also uses the detection of body movements of theoperator from muscular electrical signals to intuitively generate remoteelectrical signals. These features enable the operator to be alerted tosubtle variances in conditions which over time could become problematic.When combined with other available virtual reality technologies, theHHMI makes possible the remotality experience and control of roboticoperation as if the operator were indeed the robot rather than a remoteobserver/controller. The operator feels, sees and hears the synchronizedsensory cues that put him in the skin of the robot whether it isreal-time, recorded or virtual, or whether it is land based, water,space, macrosized, microsized, nanosized, flying, or other moving orstationary machine. The HHMI utilizes the natural electrical physiologyof the user 12 to interface the user 12 with the machine, to conveylocally generated (e.g., the user 12) or remotely received or determined(e.g., the robot) information and to control the remotely locatedmachine.

The advancement of robotic systems in many fields has been rapid andwidespread. There has not been a concurrent evolution in the interfaceof a remote human operator with the robot, beyond the joystick and videodisplay. The industry has focused on leaps forward for the main roboticsystem, while the joystick/video interface seems to have been left as agood-enough solution. Going forward, the good-enough solution of theprevious generations of human/machine interfaces will be a constraintwhen paired with the capabilities of the next generations' robot anddrone systems.

Electrical stimulation is applied through the skin on at least one ofthe arm 16 s of the pilot dependent on a desired position to be achievedby the pilot's hand and arm 16. The desired body position can be relatedto a sensed parameter, such as flex, rotation, tilt, pitch, yaw,temperature, vibration, and other detectable stresses or conditions of amechanical component (wing, fuselage, control surfaces, etc.) of theUVS. The sensed parameter could be air pressure experienced at a wingcontrol surface while maneuvering. The sensed parameter is transmittedfrom the drone (using RF or line-of-sight optical), causing acomputer-controlled neuromuscular electrical stimulation (NMES) cue(electrical stimulation) resulting in an auto-action response in thehand of the pilot feeling pressure to assume a position directly relatedto the drone's control surface. The pressure to move the hand is theresult of muscle movements caused by the NMES cue. The pilot experiencesthe sensation of resistance or pressure because of the computercontrolled electrical signals applied to the pilot's ownsensory/muscular physiology. In addition to pressure and resistance, thephysical sensation of vibrations, knocks and even scratches can beperceived as the result of subcutaneous controlled electrical signalstimulation. The muscle movements are involuntarily and automatic. Thereare no mechanical force simulators involved, although there can be.Vibration, for example, can be stimulated by both the applied electricalsignal and mechanical buzzers (or rumble packs, etc.) that can beapplied, for example, from a haptic chair (see, for example, FIG. 69) orfrom a transducer associated with one or more of the electrodes 14 ofthe HHMI sleeve. In the case of music and entertainment, for example,the transducer could deliver the vibration as low end bass notes, whileapplied electrical signal delivers the sensation of light scratchescorresponding to higher notes. Bass beats, for example, could beperceived through a knock sensation resulting from an appropriatelycontrolled electrical signal.

The hands of a human are particularly sensitive to haptic stimulation.For example, the muscles 18 that move the finger joints are in the palmand forearm 16. Muscles of the fingers can be subdivided into extrinsicand intrinsic muscles 18. The extrinsic muscles 18 are the long flexorsand extensors. They are called extrinsic because the muscle belly islocated on the forearm 16. The application of haptic sensation, such asthe haptic sensory cues, can be applied to various parts of the body,and the inventive accelerated learning system adapted to enable a widerange of applications, from remote control operation to human/machineinterface to teaching to entertainment to rehabilitation. By noting thesensitivity to stimulation of the body parts (e.g., the fingertips arevery perceptive to tactile stimulation, the application of hapticsensory cues can be selective in accordance with a desired interface,learning or entertainment enhancement. For example, the fingers (and/orthe muscles 18 controlling the fingers and/or the nerves communicationwith those muscles 18) can receive haptic stimulation in the form of apressure, vibration, electrical impulse or other stimulation.

FIG. 78 illustrates the inventive HHMI configured as a full body suitmapped to a remote drone, and including haptic, audio and video sensorycue systems, body position and electrical activity sensing systems andbrain activity sensing system.

The haptic sensory cues are generated and applied to the pilot insynchronization dependent on the time sequential data that is receivedfrom the remote drone. In addition to the time sequential data thatpertains to the haptic cues, time sequential first sensory data is alsoreceived from the remote transmitter. This time sequential first sensorydata may be, for example, video or audio data that is collected byappropriate components on the drone. A plurality of first sensory cuesare generated capable of being perceived by a pilot. The plurality offirst sensory cues are serially generated in synchronization dependenton the first sensory data. That is, for example, the sequential framesof a video displayed to the pilot replicate the visual informationcollected by camera 38 s on the drone in time sequence. The plurality offirst sensory cues are effective for stimulating at least one additionalsense of the user 12, including vision, hearing, smell and taste (inthis example, vision). The haptic sensory cues are generated insynchronization dependent on the plurality of first sensory cues. Thatis, the haptic sensory cues represent the flight conditions (e.g.,control surface orientation and air pressure, etc.) experienced by thedrone synchronized to the visual information from one or more camera 38s on the drone. One or both of the time sequential data and the timesequential first sensory data may include at least one sensed conditionthat is sensed at a location remote from the user 12. The remotetransmitter can be part of a remotely controlled vehicle, such as adrone, robot or remote vehicle. This enables, for example, the pilot tointuitively “feel” the forces on the drone while visually seeing theresults of a flight maneuver of the drone, such as a banking turn. Thissensory feedback to the pilot's control of the flight enables the pilotto have an intimate and immersive perception of the drone's flight.

For virtual or augmented reality, a full sphere of views may be madeavailable to a pilot wearing, for example, a head tracking virtualreality headset, such as the Oculus Rift. As the pilot looks right,left, up, down, for example, the movement of the pilot's head is trackedand an appropriate video scene can be generated in 3D on the virtualreality video headset. The perspective and zoom of the camera 38 imagedata can be controlled via a microprocessor running a stored computerprogram set of instructions so that the pilot may experience the visualcues as if the pilot is physically located anywhere on the drone(cockpit, tail, wingtips, etc.). Also, the collected video data can becombined with computer generated images so that the perspective viewedby the pilot can be from outside the drone. For example, the pilot canview the drone he or she is remotely controlling as if flying along sideor behind the drone.

Although this non-limiting exemplary embodiment describes haptic sensorycues combined with auditory and/or visual sensory cues, the combinationof sensory cues could be any combination of the senses perceivable by ahuman, including smell, taste, hearing, sight and touch.

The HHMI configured as a full body undergarment can be a component of avirtual reality interface that deepens the immersion for the operator bytying in real-time head and body movements to a three dimensional,perceived visual sphere. High quality, binaural, audio provided throughsound canceling headphones replicate the actual, real-time sounds thatare ambient to the remote robot.

The HHMI can be configured as a sleeve, legging, jumpsuit, coverall,jacket, trouser, cap, glove or other wearable electronic. The HHMI maybe comprised of a multilayered structure with the electrodes 14 incontact with the skin of the user 12, insulation and wiring layers, andthe sleeve covering. The layers, such as the outer covering may be, forexample, a thin, multi-axial stretchable fabric. The fabric can beelectrically insulating, and contain conductive threads, patches,coatings or inks to conduct the detected and applied electrical signals.In some of the drawings the electrodes 14 are illustrated as being onthe outside of the sleeve to show the concept of electrode size andlocation. In an exemplary embodiment, the sleeve is made from an opaqueLycra material with flexible conductive fabric electrodes 14 disposed onthe interior of the sleeve and in direct face-to-face electrical contactwith the skin on the arm 16 of the user 12. The fabric of the outercover or other layer provides sufficient compression to urge theelectrodes 14 into face-to-face electrical contact with the skin of thearm 16. In addition or alternatively, straps, bands, Velcro or othersuch mechanisms can be used for urging the electrodes 14 intoface-to-face electrical communication with the user's skin. Flexible andconductive fabrics, such as copper/polyester fabric can be used to makeelectrode patches that are highly conductive, thin and flexible. Signalcross talk, interference from or to the electronics of the HHMI may bemitigated with shielding layers separating, as necessary, the conductivepathways and electrically active components.

FIG. 78 illustrates the HHMI configured as a full body undergarment thatcan be a component of a virtual reality interface that deepens theimmersion for the operator by tying in real-time head and body movementsto a three dimensional, perceived visual sphere. High quality, binaural,audio provided through sound canceling headphones replicate the actual,real-time sounds that are ambient to the remote robot.

For example, the haptic cues cause the operator to experience wind gustsas sudden jarring movements, or unbalanced stresses on the mechanicaland control surfaces, such as experienced in a tight banking maneuver,as proportionally applied pressure or resistance to movement.

Thus, forces experienced by the robot are detected and transmitted, thenconverted to proportional electrical signals. The operator's body'sreceptors such as, nocireceptors mechanoreceptors, thermoreceptors,proprioceptors and chemical receptors, receive the computer controlledhaptic cues applied as electrical stimulation to replicate naturalsensations received by the human body through the skin, muscles andbones.

In a teaching scenario, in a general embodiment, the NMES is applied asthe generated sensory cue to the user 12 dependent on the position of abody part of a performer relative to a performance element of aperformance object with which an event is performed. In a more specificembodiment, such as simulated flight training of maimed or unmannedaerial vehicles, one or more sensory cues are computer controlled tostimulate the sense organs of the user 12 (e.g., student pilot)effective for stimulating various processing center of a brain of theuser 12 so that user 12 learns how to position his body membercorresponding to the position of the performer of the event. Sensorycues are applied to the user 12 and are dependent on a position of atleast one body member of a performer relative to a performance elementof a performance object with which an event is performed. For example,in addition to the haptic cue, audio and visual sensory cues can beapplied synchronously to the user 12 senses. The sensory cues areeffective for stimulating the various processing center of a brain ofthe user 12 so that user 12 learns how to, for example, rapidly achievethe position of a particular body member (e.g., hand on a joystick)corresponding to the position of an instructor or performer (e.g.,actual pilot) performing the event (e.g., flying an actual plane ordrone).

A key concern in the safety of aircraft flight is ensuring that thepilot maintains an appropriate understanding of the orientation of theaircraft. This is a concern both for manned aircraft flight, especiallyin Instrument Flight Rules conditions, as well as for unmanned aircraftoperations. In manned flight, even with in-cockpit aids such as anartificial horizon, pilots can still become disoriented and often maytrust their physical and proprioceptive senses as opposed to the cockpitaids.

In unmanned aircraft operations, pilots lack proprioceptive inputs andmust instead receive all information about aircraft orientation throughother means. Typically, this has been done through visual and auditoryaids on the ground stations of remote controllers, each of which, ifoverused, can actually become a detriment to a pilot's awareness of thesituation.

Haptic feedback provides an additional outlet for alerting the pilot tothe true state of the aircraft, but historically, haptic interfaces havenot been well-received. If improvements in haptic stimuli could reachthe point that gentle, finely-located “pressures” could be applied tothe pilot's body in varying locations (to promote a sense of beingupside down or tilted to the side), it may provide an additionalalerting mechanism to inappropriate aircraft orientations. In addition,a variety of other alerts could potentially be sent through a similarinterface.

FIG. 79 illustrates the inventive HHMI configured for applyingelectrical stimulation to large muscle groups 18 to provide haptic cuesof a maimed or unmanned aerial vehicle. As described herein, the HHMIprovides such an interface with haptic feedback in a product platformthat can be integrated into the existing and future robotic systems.Data from sensors on a remote UAV is used to indicate remote flightconditions via an electrical stimulation (haptic cues) applied to thepilot (e.g., warning the pilot that the drone is in an unintendedbanking condition or is approaching an obstacle). The sensors of thedrone are mapped to the operator so that the large muscle groups 18 ofthe back, shoulders and thighs become indicators to the operator of theflight conditions of the drone. A similar system can be used for pilotsof aircraft, long haul truckers and others where an indication ofimportant information can be of benefit and conveyed via the HHMI. Forexample, a pilot may be made aware of a subtle banking that is puttinghim off course but might otherwise not be noticed. A long haul truckercan be woken up if she falls asleep behind the wheel.

The HHMI is also an enabling technology for remotality that is to beintegrated in a variety of existing and future products, not just forunmanned aerial systems and other robotic systems and the various otherapplications and uses described herein. The elements, construction,circuits, apparatus, computer programming code, methods and algorithmsdescribed herein with reference to the various exemplary embodiments maybe employable as appropriate to other uses of the HHMI, some of whichare described herein, other of which will be readily apparent whendescribed and inherent features of the HHMI are considered.

As shown, the inventive human/machine interface includes a virtualreality visual system that deepens the immersion for the user 12 (e.g.,pilot, student, sports fan, patient, etc.) by tying in real-time headand body movements to a three dimensional, perceived visual sphere.Camera systems onboard the drone feed real-time video from enough camera38 angles to create a seamless (after software cleanup) sphere ofvision. As an example, if the pilot is sitting in the inventive chairshown in FIG. 69, this virtual visual sphere could give the pilot theimpression that he is flying a glass chair rather than a drone.

The audio system of the ALS may include high quality, binaural, audioprovided through sound canceling headphones to replicate the actual,real-time sounds that are ambient to the remote UVS, or other soundssuch as white noise, soothing or aggressive music, depending on theintended mood and temperament of the user 12. The inventivehuman/machine interface is intended to isolate the pilot from localambient distractions, but the degree of this isolation can be easilycontrolled to maintain safe conditions. Also, although a bit moreinvasive than surface electrodes 14, the electrodes 14 used to apply ordetect the electrical signals can be of a type where the skin ispierced. However, piercing the skin is not necessary to effect benefitsfrom the inventive human/machine interface and much less invasive gels,gel electrodes 14, carbon fiber electrodes 14, etc., can be used.

In addition to the full immersion of visual and auditory stimulationcorresponding to the remote drone as it flies, the application ofauto-action and other haptic cues enable the pilot, in a sense, tointimately “feel” the flight conditions experienced by the remote UVS.With the level of remotality immersion into the real-time conditions ofthe UVS created by the inventive human/machine interface, the pilot doesnot just feel like he or she is flying the UVS, to the extent possible,the pilot becomes the UVS.

The HHMI configured as a tactile suit can treat autism by providing asensation replicating light pressure thereby providing therapeuticbenefits using a custom-calibrated, mobile and convenient system.

FIG. 80 shows the plurality of drones having proximity sensors fordetecting other drones, ground, and other potential collision obstacles.A formation of such VR HHMI enabled drones could be effectively pilotedby people anywhere in the world, all experiencing simultaneousside-by-side flight. The pilots would feel the presence of their flyingneighbors, with head tracking VR goggles and binaural audio thatcompletes the immersion. Such as system may be useful, for example, forsearch and rescue in remote areas, even expanding the volunteer base farbeyond the local area.

The drones could be a commercial package delivery service flyingautonomously in formation along defined skyways. Then, “pick up” a humanVR pilot to safely complete the flight from the skyway to the door steppackage delivery. The human is only flying the drone when needed, whenmore things could go wrong and the agility of the VR HHMI would be mostuseful.

The data from proximity sensors can be used to allow drones to fly information for example, when delivering packages from a central warehouseto a neighbor hood. The formation flying can be achieved with anoperator providing flight control for a squadron of drones so that costof having a human operator controlling each drone during the commonformation flight is avoided. The squadron of drones maintain safeformation by each drone knowing and responding to proximity details muchas a real-world flock of birds or school of fish achieves theirseemingly impossibly timed individual maneuvers. Once the neighbor hoodhas been reached, the flight of each drone can be take over or monitoredby a human operator to safely deliver the package. Once the packaged isdelivered, the drones may for up in formation again and again achieve asafe flight back to the warehouse under the control of a signal remoteoperator.

The proximity signals can also be applied to the drone operator formilitary, gaming or entertainment purposes, such as for an aerial combatbetween drones. In addition to the proximity data being applied ashaptic cues, for example, other haptic cues may cause the pilot toexperience wind gusts as sudden jarring movements, or unbalancedstresses on the mechanical and control surfaces, such as experienced ina tight banking maneuver, as proportionally applied pressure orresistance to movement. Even subtle nuances such as the warm 16th of thesun shining on the top surfaces of the drone can be experienced at acorresponding location on the back of the pilot. It isn't yet known whatdegree of immersion and which nuances might be optimal for a given setof circumstances, the inventive human/machine interface is designed withthe intention of enabling the high quality resolution of multiplecomputer generated, enhanced and real-time synchronously applied,immersive sensory cues.

Thus, forces experienced, for example, by the drone, are detected andtransmitted, then converted to proportional electrical signals. Thepilot's body's sensation receptors such as, nocireceptorsmechanoreceptors, and thermoreceptors including proprioceptors andchemical receptors, receive the computer controlled haptic cues appliedas electrical stimulation to replicate, for example, natural sensationsreceived by the human body through the skin, muscles 18 and bones. Sincethe nervous system of the human body operates via electrical impulses,any nerve, nerve ending, muscle or receptor can be triggered byelectrical stimulation. Signal characteristics such as; the location,timing, pulse length, frequency and amplitude of the electricalstimulation are applied under the control of the computer depending onthe intended type of sensation or muscle movement to indicate to thepilot the drone's onboard and ambient conditions.

Depending on the applied NMES cue, the pilot experiences the haptic cueas pressure, as if pushing against resistance and/or being forced tomove into the position related to the wing control surface, and/or avibration or even a blow as if being jarred by an external force (e.g.,being buffeted from a wind gust).

The inventive human/machine interface has an advanced multi-sensorysystem that uses the physiology of the pilot to integrate the onboardand ambient conditions of a remotely flown drone into the informationpool used by the pilot to control the drone's flight.

As shown and as described herein throughout, the inventive HHMI adds anew mode of interaction to the human/machine interface. Haptic sensoryfeedback is used to provide touch cues related to the remote robotinternal and ambient conditions. Also, using the same basic structure,the detection of body movements of the operator from muscular electricalsignals can be utilized to generate remote electrical signals. The HHMI,when combined with available virtual reality products and proprietarysoftware and hardware, completes the immersion so that the brain of theoperator processes the received haptic, visual and auditory cues for anew form of robotic teleoperation and telepresence.

Remote Surgery

The HHMI can be employed to provide haptic feedback to a surgeonperforming robotic surgery. FIG. 82 is a configuration of the inventiveHHMI configured for robotic surgery. For example, at a location remoteto the surgeon, a robotic surgical station can use, for example, ascalpel to cut through the skin of a patient. The robotic surgicalstation is capable of operating under the control of the remotelylocated surgeon. However, using a typical remote surgery system, theremotely located surgeon is relegated to viewing the skin being cut on atwo dimensional video screen and typically receives little or nofeedback related to the mechanical forces that are involved in cuttingthrough the skin. The surgeon does not feel the initial resistance tothe tip of the scalpel as the skin is cut, and then the quick change inresistance once the skin has been penetrated. This can cause overpenetration and too deep of a cut, especially when performing delicateoperations involving reconstruction of skin such as facial surgeries orwhen removing a tumor from a highly sensitive area where nerves, bloodvessels or other delicate body constituents are present. Skinreconstruction is just an example operation that benefits from thetactile feedback and VR/AR remotality experience enabled in accordancewith this use of the HHMI.

In accordance with this aspect of the invention, the remote surgeon canview the surgery using head tracking 3D goggles, hear the ambient soundsof the operating room, and feel the mechanical forces involved in thesurgery.

In accordance with this aspect of the invention, a force is detectedresulting from a robotic surgical operation, such as the resistance tothe scalpel tip at the beginning of the cut, followed by less resistanceonce the skin has been penetrated and is being cleanly sliced. Atransmittable signal is generated dependent on the detected force.Following this example, the detected force can be pressure needed toovercome the resistance by the skin to the scalpel and the pressure ofrapid change in resistance once the skin has been penetrated, which inan non-remote surgical procedure would provide direct tactile feedbackto the surgeon to immediately ease up on the pressure exerted throughthe scalpel tip and edge to the patient's body. The transmittable signalis by an appliance in physical communication with a body part of a user12 performing the robotic surgical procedure. For example, the appliancecan be a remotely controlled robotic arm holding the scalpel. The remotesurgeon may hold a stylus that has a weight and shape consistent with asurgical scalpel. The stylus may be in contact with a variable viscositymaterial (such as an electrical rheological fluid) so that there is aphysical structure in contact with the stylus that resists the movementof the tip of the scalpel in proportion to the detected skin resistance.Alternatively, or additionally at the same time when the scalpel ismaking the actual cutting, haptic cues can be applied to the remotesurgeon that are dependent on the received transmittable signal. Thecombination of the variable viscosity physical structure and the HHMIapplied haptic cues (e.g., involuntary movement and somatic/kinestheticsensations) provides valuable feedback to the surgeon so that a safer,more effective remote surgical procedure can be performed.

Immersive Sporting Event Observation and Participation

One can envision a virtual sporting event where the players are locatedgeographically remote from each other, as shown in FIG. 75, and observeand participate in the sport as shown in FIG. 83-54, using theremotality enabled by the HHMI, AR and VR systems described herein. FIG.83 shows a sport apparatus configured as sensory data detecting shoulderpads worn by a football player. FIG. 84 shows a sport apparatusconfigured as a sensory data sensing helmet worn by a football player.The shoulder pads, helmet and other gear, including an HHMI configuredto be worn as a shirt or full body undergarment by a player, providedata that is sensed during the on-field game play. The sensory data iscollected from the perspective of the player. FIG. 85 illustratessensory data detected by an on-field player, where the data is appliedas sensory cues for virtual immersion by a football fan. The sensory(haptic) data detected by an accelerometer, pressure and othertransducers embedded in pads, gear and clothing, along with the audioand visual data collected from helmet can be transferred to a remotelylocated fan wearing an HHMI and associated VR/AR equipment.

For example, the locations and relative strength of force felt by theball carrying player on the field in the grips of opposing side tacklerscan be mapped to an HHMI worn by the sports fan and proportionallysensed by the fan as somatic and/or kinesthetic sensations. Similarly,these force data can be mapped to the haptic chair and vibrations and/orpressures exerted to the body of the fan while viewing the play from theball carrying player's perspective. The HHMI can be configured to looklike the jersey worn by the fan's favorite team. If more than one playerhas the data determining equipment, such as video, auditory and haptictransducers, then the fan can switch around and experience the game fromthe perspective of multiple players.

The audio and haptic sensory cues provided by haptic chair may bebinaural sound, vibration, electrical signals, etc. that are dependenton the haptic and audio data detected from the player on the field. Thehaptic chair can provide sensory cues responding to on field data fromplayer so that a fan experiences haptic sensations that are synchronizedto a 3D head tracking image and binaural sound from the perspective ofthe player on the field (using cameras 38 and microphones 36 embedded,for example, in the helmet worn by the player). FIG. 86 shows sportsapparatus configured as sensory data sensing helmet, glove, baseball andbat used by a baseball player. Haptic, visual and audio data detected bysensors on gloves, helmet, bat, ball, etc. is transmitted to fan forvirtual immersion in a manner similar to the football example describedabove. The haptic human/machine interface (HHMI) can be used toalleviate disabling motor systems of Parkinson's Disease (PD). The HHMIsleeve provides transcutaneous electrical activity detection of themuscles and nerves involved in oscillating limb movement of a PD tremor.In response to the detected tremor, the HHMI sleeve applies anelectrical signal to cause involuntary motor unit impulses thatcounteract the undesirable trembling that would have otherwise occurred.Among the advantages of this approach is on-demand, immediate andautomatic relief without drugs or surgery.

Treatment of Movement Disorders

The pharmacological treatment of tremor often has undesired sideeffects. For example, Levodopa is a potent drug for controlling PDsymptoms. However, over time levodopa frequently results in motorcomplications, such as fluctuations and dyskinesias, making the use ofhigher dosages of levodopa a difficult decision, especially forrelatively younger patients. Although deep brain stimulation has beenshown to be more effective than the pharmacological approach fortreating tremor, it is a highly invasive surgical procedure that isassociated with an increased risk of serious adverse events.

Physiological tremor is measured in healthy individuals as a lowamplitude postural tremor with a modal frequency of 8-12 Hz. Thephysiological tremor measured in a healthy individual at rest is anormally occurring, low amplitude oscillation determined by mechanicallimb properties and cardioballistics. The degree of regularity, measuredby approximate entropy, in the limb acceleration signal (measuredmovement data) and the coherence between limb acceleration and muscleoutput (measured EMG data) has also been shown to be useful incharacterizing both physiological and pathological tremors.

EMG measurements have been demonstrated to be reliable predictors ofmotor unit synchronization in tremors. Studies have shown that evenunder varying loads on the tremor-effected limb, the dominant tremorrate of PD tremors as measured by EMG showed constant frequencyreflected by corresponding sharp peaks in the EMG spectra irrespectiveof changes in mechanical resonant frequency.

The HHMI makes use of this ability to determine tremor musclecontractions using EMG and movement sensors, and is designed to adapt tothe physiology of different patients by automatically determining thedominant tremor muscles by analyzing the detected signals received froma number of electrodes located at the various muscles of the patientsforearm. Tremor in PD is classified as a resting and/or postural tremor,which also has increased amplitude, regularity, and tremor-EMG coherence10 Hz. The modal frequency of resting tremor in PD is between 3 and 5 Hzand postural tremor frequency ranges from 4 to 12 Hz. In addition todetermining the dominant tremor muscles, the HHMI also analysis the EMGand movement data and makes adjustments to the application of themitigation signal so that changes in tremor frequency and strength isautomatically accommodated.

The HHMI can be used in conjunction with traditional therapeuticchoices, and may reduce the dosage or aggressiveness of the drugconventionally used to treat tremor, and may obviate or at least delaythe need for invasive deep brain surgery.

The use of transcutaneous electrical nerve stimulation has been approvedby the FDA for pain relief, and more recently for preventing migraineheadaches. In accordance with an embodiment, the detection side of theHHMI uses an EMG system, such as the DelSys Trigno wireless sensorsystem and EMGworks® Acquisition and Analysis software. Computeranalysis of the detected EMG and movement data is done on the outputfrom the EMGworks® software, and/or from the analog-out of the Trignobase station, integrated with our HHMI proprietary software running onan Arduino YUN microprocessor.

For signal generation and application, the HHMI multiplex electroniccircuit shown in FIGS. 34-42, for example, works with the microprocessorto create a selectable array of electrodes. The multiplex electroniccircuit adds further adjustability to the waveform of the appliedsignal, and iterations of this circuit are explored using, for example,pulse width modulation, passive/active matrix electrode selection, AC,DC pulses, and high-speed transistor switching. The microprocessorcontrols signal switching elements of the multiplex circuit to determinewhich of the individually addressable electrodes apply the differentTENs signals (or no signal). The EMG detection electrodes are alsoindividually addressable through the multiplex circuit. As an example,with the high speed switching multiplex circuit, very high sample andsignal application rates can be achieved allowing a large number ofindividually addressable electrodes to selectively detect and applysignals.

Parameters such as type, number, size, and shape of the electrodes canbe varied. The range, timing, sampling rate, etc., of the detected andapplied signals can be semi-automatically and automatically adjusted,with an onscreen graphical user interface for adjusting the digitallycontrolled electronic circuit elements and generated signalcharacteristics.

Design flexibility can include using wireless Bluetooth connectivity toallow, if necessary or beneficial, the off-loading of parts of thereal-time signal analysis to a more powerful desktop or networkedcomputer, with local and/or cloud storage. An efficient chip-baseddetecting system can be constructed, for example, working with Neurosky(San Jose, Calif.) and their BMD101 biosensor chip. A chip based sensorenables local amplification the weak signal close to its source, usingthe chip's analog front end to condition the weak signal to achieve amore easy to process SNR. A built in digital processor in the chip canemployed to off-load much of the number crunching that would otherwisehave to be done by the general purpose microprocessor.

Dry electrodes, such as the examples shown in FIGS. 47-54 can be usedfor detection and application of electrical signals. Dry electrodes aremore convenient, comfortable and much longer lasting than gelelectrodes, and therefore more suited to a wearable electronic. Aversion of the dry electrodes may be constructed with a concentricseparated signal detector/signal applier electrode configurationfabricated by saturating the surface of a conventional gel electrodewith microscopic conductive beads to form a hybrid dry/gel electrode.Another electrode configuration includes a rubber bristle structurefabricated using soft polymer bristles that are electroplated withAg/AgCl. The bristles may be thin and flexible, terminating in aflattened skin contact area to maximize the face-to-face electricalcommunication between skin and bristle gang. As part of the HHMI sleeve,the spaced bristle structure also promotes airflow at the skin surface.The Ag/AgCl electrical contact optimizes the ionic conduction for EMGdetection and TENS signal application. As an example, to make thebristles conductive they can be coated with a conductive ink, silver iselectroplated on the conductive ink and then anodized in a solution ofKCl or similar salt to form an adherent AgCl coating on the silver.

Dry electrodes have higher contact impedances as compared to gelelectrodes. For example, because there is such a weak EMG signalavailable for transcutaneous detection, the a very short distance and/orFaraday cage-type shielding of the conductor between the electrode(where the signal is received) and the first amplification of thereceived signal may be used. If there is too much noise or crosstalkinterference, active electronic elements to amplify the EMG signal veryclose to the source (skin surface) will be built directly into the dryelectrode pad. Real-time impedance monitoring of the quality of the dryelectrode contact enhances the ability to make adjustments as necessaryto the electrodes and the conduction path of the detected and appliedsignals. If the detected signal from the dry electrodes has too muchnoise or crosstalk and is not adequate for real-time analysis, the HHMIsoftware and hardware can include known elements/techniques to obtainlow latency between tremor EMG signal detection, mitigation signaldetermination and TENS signal application.

Movement disorders are chronic, often painful, and debilitatingconditions that affect the ability to control movement. Having amovement disorder can make it difficult—even impossible—to do theroutine things in life. More than 40 million Americans—nearly one inseven people—are affected by a movement disorder, including tremor,Parkinson's disease, Tourette's syndrome, dystonia, and spasticity TheHHMI can be used in combination with a core-steadying gyroscope shown inFIGS. 87 and 88 to be a wearable electronic for treatment of balance andgait disorders. Vision, the vestibular system and the somatosensorysystem all work harmoniously to maintain posture and balance in ahealthy individual. Vision is primarily used in movement planning andavoiding obstacles. The vestibular system senses linear and angularaccelerations, and the many sensors of the somatosensory system are usedto determine body part position, contact and orientation.

The HHMI can be configured in combination with augmented reality (AR)and virtual reality (VR) vision and auditory systems for use as awearable electronic for cognitive therapy. The HHMI may be used forexploring disease trends and diagnosis using advancements incloud-storage, and the analysis of Big Data with artificial intelligence(network-connected HHMIs can upload anonymous and encrypted collectedmedical information from potentially millions of patients).

The HHMI combined with recently available Augmented and Virtual Realitysystems provide computer-controlled sensory cues (haptic, audio, andvisual) applied to “re-calibrate” or “re-wire” the brain and nervoussystem through the simultaneous application of synchronized sensorycues. The cognitive therapies can take advantage of neuroplasticity andrebuild damaged processing capabilities that are the cause of movementdisorders' physical symptoms. There is growing evidence that learning anew skill such as playing the piano, can be an effective tool forrehabilitation. The practicing and playing of a musical instrumentreinforces the association of motor actions with specific sound andvisual patterns (musical notation) while receiving continuousmulti-sensory feedback. The connections between auditory and motorregions (e.g., arcuate fasciculus of the brain's frontal lobe) arestrengthened while multimodal integration regions (e.g., around theintraparietal sulcus of the brain's parietal lobe) are activated. Avirtual cognitive experience can be created that is composed ofsimultaneously applied sensory cues that stimulate the touch, hearingand visual processing centers of the subject's brain. The visual sensorycues of the experience will be seen by the subject through the VR/ARheadset. The audio sensory cues will be heard through high qualitybinaural headphones. The touch sensory cues are applied through the HHMIgarment.

Although exemplary embodiments are adaptable to all or partial computergeneration of the sensory cues (e.g., similar to a video game), recordedor real-time sensory information can be obtained from a performer, suchas a pianist, interacting with a physical object, such as a piano. Inthis case, the HHMI and other hardware/software components are used todetect the hand and finger positions of a pianist. A head-mounted videocamera, such as a Nikon Keymission 360, can record the piano playingfrom the pianist's perspective, and high quality audio will be recorded,again from the pianist's perspective, using a Freespace BinauralMicrophone. The haptic, audio and visual sensory information is thusobtained using the EMG and motion detection capabilities of the HHMIcombined with available pattern recognition software and hardware (e.g.,Leap Motion). In this example, the actual movements of a real pianistare used to determine the sensory cues that will be applied in an effortto teach piano to the subject. A cognitive therapy session can occureither remotely or at the instrument. The HHMI can be used for cognitivetherapy with the subject at the piano, and also while sitting in acomfortable easy chair. In either case, the combination of haptic, audioand visual cues are applied to reinforce the learning of the piece ofmusic, and rewiring damaged neuronal communication links.

As shown in FIGS. 87 and 88, the HHMI can be configured with a Gyro-Vestthat holds a core-stabilizing gyroscope adjacent to the chest of thewearer. A review of the literature indicates that the swaying of ahealthy individual while maintaining balance can be modeled as aninverted pendulum. The HHMI uses a detectable EMG signal thatcorresponds to the muscle groups that are activated to maintain theinverted pendulum sway (i.e., the BCMs), and that up to a point(determined by the spinning gyroscope mass and rotational speed), theswaying of the torso core will undergo proportional inertial resistancefrom the gyroscope. The HHMI detects EMG, movement and inertia data atthe limbs, BCMs and torso core. Using the HHMI Torso Suit shown in FIGS.87 and 88, the Balance Control Muscles (BCMs) of a test subject aredetermined and involuntarily activated in conjunction with acore-steadying Gyro-Vest. The HHMI is applicable, among other uses, tocognitive therapy, accelerated learning, brain/spinal cordrehabilitation, balance restoration and tremor mitigation.

The HHMI can be configured for an interface between a human operator anda remote machine where inertia changes of the subject wearing the HHMIdetect Movement Disorder Motion (MDM). Electromyography detects theactual MDM causing muscles. The detected information is digitallyanalyzed to determine MDM-opposing muscles and MDM mitigation signals.The MDM mitigation signals are applied to stimulate the MDM-opposingmuscles. The target for detection is the muscle/nerve motor unitsresulting in MDM, with the mechanism of action for mitigation beingMDM-opposing muscle contractions that restrain the MDM.

Remote Control and Sensing

The advancement of unmanned vehicle systems for many applications hasbeen rapid and widespread. There has not been a concurrent evolution inthe interface of a remote human operator with the unmanned vehicle,beyond the joystick and video display. The industry has focused on leapsforward for the main robotic system, while the joystick/video interfaceseems to have been left as a good-enough solution. Going forward, thegood-enough solution of the previous generations of robotics will be aconstraint when paired with the next generations.

A haptic interface uses touch and movement to allow a human to interactwith a computer. The interaction can be through bodily sensations, wherecomputer-controlled touch sensory cues are received as the input to thehuman, and detected movement, where the determined movement of the humanbody is received as input to the computer.

UVS operators and the UVs they control are typically remotely locatedfrom each other. This remoteness creates human factors issues. Operatorsof UAVs note a number of interface issues physical arrangement of thedisplays (too far apart), the unnecessarily complicated retaskingprocesses, and difficult-to-read displays.

Looking back at least several UVS technology generations ago, criticalprinciples of aviation display design was developed through anunderstanding of the psychology of information processing. Alerts areanother aspect of the human/machine interface that should be carefullyconsidered. It is important for alerts to be easily interpreted. Alerts,visual, auditory, or otherwise, must signal to operators if there is asituation that requires attention, but ideally with minimal work-flowdisruption.

There is already a lot of multisensory information provided to theremote operators to convey the status of the unmanned vehicle. Onboardsensors and cameras can provide real-time visual, kinaesthetic,vestibular and auditory information. Most of this information isconveyed using instruments and displays monitored by the remoteoperator. Even with the best display technologies andgraphical-user-interfaces, the status and environment of the UVS isdifficult to absorb.

With conventional interfaces, remote operators do not have adequatemulti-sensory cues, like one has in a manned vehicle, to reallycomprehend changing conditions such as orientation, speed, impacts andvibrations. The kinaesthetic cues available from the motion of a mannedvehicle are not available to the remote operator to give an ambientsense of movement and gravitational forces. For example, the operator ofa UAV has to rely on a visual or audio warning for a condition that thepilot of a manned aircraft would detect through non-visual/audio motionand touch senses. The pilot knows by proprioception, by the internalsense of the relative position of the body's musculoskeletal system andresistance to movement, when a sudden banking occurs due to a gust awind. The lack of adequate sensory cues available to the remote operatorcan result in a failure to detect or correctly diagnose a problem,compromising the safe and effective control of the vehicle, orjeopardizing the mission.

The Multiple Resource Theory holds that different sensory modalitiesdraw from different attentional resources. It has long been suggested tomake multisensory cues available to the operators of remote vehicles. Ithas been suggested that a multisensory interface reduces the overload onan operator experienced when a particular sensory mode, for example,vision, is swamped with information.

Over four decades ago, Professor Steve Mann, who has been called thefather of wearable computing, foresaw a new approach to thecomputer/human interface where the apparatus is always ready for usebecause it is worn like clothing. This led to the evolution of a conceptMann calls Humanistic Intelligence where video and audio sensory cuescreate a “perceptual reality mediator.” This generalized mediatedperception system can include deliberately induced synesthesia whereneurological stimulation of one sensory or cognitive pathway leads toautomatic, involuntary experiences in a second sensory or cognitivepathway.

Humanistic Intelligence theory thinks of the wearer and the computerwith its associated input and output facilities not as separateentities, but regards the computer as a second brain and its sensorymodalities as additional senses, in which synthetic synesthesia mergeswith the wearer's senses. In this context, wearable computing has beenreferred to as a “Sixth Sense”.

Now, due in large part to Moore's Law and the continuous miniaturizationof electronics, and other technologies, such as small, lightweight,ultra high resolution displays, Dr. Mann's decades-long vision forHumanistic Intelligence and wearable computing will soon be as commonplace as the ubiquitous cellphone.

It is hard to doubt that a multi sensory human/machine interface wouldoffer significant advantage for all kinds of uses of remotely controlledunmanned systems.

The enabling technologies for a truly Deep Immersion Human/MachineInterface now exist and soon, in an artificial but still veryperceivably real sense, the operator will “become” the unmanned vehicle.

The HHMI brings the “sixth sense” of Humanistic Intelligence to theforefront of Unmanned Vehicle Systems by combining wearable computing,an immersive haptic interface and augmented/virtual reality.

There have been envisioned wearable computing systems that use theinputs from biosensors located on the body as a multidimensional featurevector with which to classify content as important or unimportant. Inthe context of the HHMI/AR/VR system described herein, sensed EMG andmovement data is provided as input to a microprocessor. The data isanalyzed and used to control a remote UV, while audio, video and hapticinformation related to onboard UV conditions is provided to theoperator.

The HHMI is configured in combination with augmented reality (AR) andvirtual reality (VR) vision and auditory systems to create a wearableelectronic for unmanned vehicle systems. The HHMI providestranscutaneous electrical activity detection of the muscles and nervesinvolved in the intended control movements of the remote controloperator. In response to the detected control movements, control signalsare transmitted to the remote unmanned vehicle. Sensed information fromthe unmanned vehicle is in turn received by the HHMI and applied as anelectrical signal to cause involuntary motor unit impulses and simulatedskin and proprioception sensations that counteract the undesirabletrembling that would have otherwise occurred.

The HHMI creates a computer-controlled artificially created sense orperception that replicates actual proprioception sensations at asubconscious level. This artificially created perception is furtherconfirmed by the augmented and virtual reality visuals creating anultra-deep immersion that is experienced as an altered reality. Thesense of proprioception is gained primarily from input from sensorynerve terminals in muscles combined with input from the body'svestibular apparatus. The HHMI stimulates the nerves and muscles toproduce a haptic sensory experience that is directly related to theremote ambient environment and sensed conditions of the UV.

The HHMI maximizes information placement and prioritization on anaugmented reality headset where the information is superimposed data onthe actual visual scene of the user (allowing for interactions in thereal-world, more immediately accessible, better for short range unsafeconditions such as battlefield and first responders). The HHMI can alsobe used with a virtual reality headset to create total immersion wherecomputer-controlled interactions can place anything anywhere. With thebinaural features of the HHMI, sound can be used to focus the pilotsvisual attention in the virtual space towards a certain direction. Thehaptic and biosensing features of the HHMI allow predefined, intuitivemovements to determine the operator's remote control intentions, and toapply involuntary movements and simulated sensations to convey real-timeflight conditions to the operator.

The HHMI adds a novel dimension to the human/machine interface. The HHMIutilizes haptic sensory feedback creating relevant touch cues related tothe remote UV onboard/ambient conditions, and the detection of bodymovements of the operator from muscular electrical signals tointuitively generate remote control signals. These features enable theoperator to be alerted to subtle variances in conditions which over timecould become problematic. When combined with the recently availableaugmented and virtual reality technologies, the HHMI makes possible theexperience and control of UV operation as if the operator were indeedthe UV rather than a remote observer/controller. The operator feels,sees and hears the synchronized sensory cues that put him in the skin ofthe UV.

Augmented and virtual reality headsets, and binaural headphones, arepaired with the HHMI to enable varying degrees of altered realityimmersion, enabling the control of a remote unmanned vehicle system andcreating an experience as if the pilot were onboard the unmanned vehiclesystem. In fact, an altered reality experience can be created where thepilot experiences the remote vehicle control as if he is the vehicle.

A key concern in the safety of aircraft flight is ensuring that thepilot maintains an appropriate understanding of the orientation of theaircraft. This is a concern both for maimed aircraft flight, especiallyin IFR conditions, as well as for unmanned aircraft operations. Inmaimed flight, even with in-cockpit aids such as an artificial horizon,pilots can still become disoriented and often may trust their physicaland proprioceptive senses as opposed to the cockpit aids. In unmannedaircraft operations, the remote pilot lacks proprioceptive inputs andmust instead receive all information about aircraft orientation throughother means. Typically, this has been done through visual and auditoryaids on the ground stations of remote controllers, each of which—ifoverused—can actually become a detriment to a pilot's awareness of thesituation.

Haptic feedback provides an additional input for alerting the pilot tothe true state of the aircraft, but historically, haptic interfaces havenot been well-received. If improvements in haptic stimuli could beimproved to the point that gentle, finely-located “pressures” could beapplied to the pilots body in varying locations (to promote a sense ofbeing upside down or tilted to the side), it may provide an additionalalerting mechanism to inappropriate aircraft orientations. In addition,a variety of other alerts could potentially be sent through a similarinterface.

The HHMI is combined with recently available Augmented and VirtualReality systems to explore computer-controlled sensory cues (haptic,audio, and visual) applied in an attempt to “re-calibrate” or “re-wire”the brain and nervous system through the simultaneous application ofsynchronized sensory cues. For example, the HHMI can be paired with theOculus Rift virtual reality headset or with the Meta augmented realityheadset.

In a teaching mode, the HHMI VR/AR configuration takes advantage of thebrain's neuroplasticity to build muscle and pattern memories that arethe “best practices” of top remote control pilots. The HHMI VR/ARconfiguration is usable in an accelerated learning system to teachnovice pilots from the perspective of a top gun. In this flightsimulation mode, the HHMI is used to reinforce the association of motoractions with specific sound and visual patterns corresponding to thecontrol of an unmanned vehicle, while receiving continuous multi-sensoryfeedback. The connections between auditory and motor regions (e.g.,arcuate fasciculus of the brain's frontal lobe) are strengthened whilemultimodal integration regions (e.g., around the intraparietal sulcus ofthe brain's parietal lobe) are activated. In this mode, the HHMI is atool for exploring the use of a wearable electronic with the latestadvancements in virtual reality for creating the ultimate flightsimulation.

In this flight simulation mode, the HHMI and VR/AR configuration createsa virtual cognitive experience composed of simultaneously appliedsensory cues that stimulate the touch, hearing and visual processingcenters of the subject's brain. The visual sensory cues of theexperience will be seen by the subject through the VR/AR headset. Theaudio sensory cues will be heard through high quality binauralheadphones. The touch sensory cues are applied through the HHMI garment.

Although the flight simulator mode is adaptable to all or partialcomputer generation of the sensory cues (e.g., similar to a video game),the recorded sensory information can be obtained directly from an actualflight with the nuances of control movements from a performer, such asexpert remote control operator. In this case, the HHMI and otherhardware/software components are used to detect, for example, the handand finger positions of the pilot expert on the control interface of theunmanned vehicle system.

The HHMI can be used with an artificial real-time perspective view of aUAV as displayed on a visual cue system such as an augmented or virtualreality headset. Data from onboard cameras and onboard accelerometers,GPS, etc., plus stored image data of the UVS are used to create areal-time artificial perspective of the UV in flight that is received bythe remote pilot as the visual sensory cues. The UV and the scene aroundthe UV can appear to the pilot from any visual perspective.

The electrical signals applied by HHMI invoke a perception by the pilotrelated to the sense of touch. These received haptic sensory cues can beapplied as computer-controlled electrical signals that are mapped to thebody of the pilot so that different body parts receive different sensorystimulation. For example, the hands and arms of the pilot may beconsidered the human embodiment of the control surfaces, such as flapsof a drone UV. The feet and legs of the pilot may be considered thehuman embodiment of propulsion components, such as the engines of thedrone. In this example, the flexing of one or both feet of the pilot canbe detected and converted to flight control signals to control theengine speed (and thereby control the speed of the drone). Engine speeddata received from the drone can be converted into a haptic sensory cuethat is displayed along with visual speed data, such as GPS determinedspeed relative to ground, so that the pilot has an intuitive sense ofthe drone's engine speed (for example, intensity of a sensed vibrationcan be correlated with the RPM of the engine) and along with the visualconfirmation of the drone's speed relative to ground. The pilot receivesmultiple sensory cues that are inter-related and synchronized toindicate the flight conditions of the remote UV.

This enables, for example, the pilot to intuitively “feel” the forces onthe UV while visually seeing the results of a flight maneuver of the UV,such as a banking turn. This sensory feedback to the pilot's control ofthe flight enables the pilot to have an intimate and immersiveperception of the UV's flight.

A 360 degree camera system collects video information onboard the UV.The placement and number of cameras enable a full sphere of viewsavailable to a pilot wearing a head tracking virtual or augmentedreality headset. The perspective and zoom of the camera image data canbe controlled via software so that the pilot may experience the visualcues as if the pilot is physically located anywhere in or on the UV(cockpit, tail, wingtips, etc.).

The HHMI is intended to provide such haptic feedback in a productplatform that can be integrated into the existing and future roboticsystems. The HHMI represents a possible paradigm shift in the interface,and therefore the utility, of the robotic system.

The HHMI adds a new mode of interaction to the human/machine interface.Haptic sensory feedback is used to provide touch cues related to theremote robot onboard and ambient conditions. Also, using the same basicstructure, the detection of body movements of the operator from muscularelectrical signals can be utilized to generate remote control signals.The HHMI, when combined with available virtual reality products andproprietary software and hardware, completes the immersion so that thebrain of the operator processes the received haptic, visual and auditorycues for a new form of robotic teleoperation and telepresence.

Computer-controlled electrical signals are applied with signalcharacteristics effective to stimulate one or more of the tactilereceptors found in the skin. The signal characteristics are controlledto selectively stimulate the receptors that have, for example, differentreceptive fields (1-1000 mm2) and frequency ranges (0.4-800 Hz). Forexample, broad receptive-field receptors like the Pacinian corpuscleproduce vibration and tickle sensations. Small field receptors such asthe Merkel's cells, produce pressure sensations. Flight condition datais used to determine the characteristics of the computer-generatedelectrical activity applied through the HHMI to causes a desiredautomatic and involuntary movement of the operator's body parts. Theresult is the operator perceives the involuntary movement as thoughcaused by an externally applied force, for example, as through theoperator's hand is the flight control surface deflecting air.

The HHMI can be configured as a full body suit that can be a componentof a virtual reality interface that deepens the immersion for theoperator by tying in real-time head and body movements to a threedimensional, perceived visual sphere. High quality, binaural, audioprovided through sound canceling headphones replicate the actual,real-time sounds that are ambient to the remote UV.

For example, the haptic cues cause the operator to experience wind gustsas sudden jarring movements. Unbalanced stresses on the mechanical andcontrol surfaces, such as experienced in a tight banking maneuver, areexperienced as proportionally applied pressure or resistance tomovement.

Thus, forces experienced by the UV are detected and transmitted, thenconverted to proportional electrical signals. The operator's body'sreceptors such as, nocireceptors mechanoreceptors, and thermoreceptorsincluding proprioceptors, receive the computer-controlled haptic cuesapplied as electrical stimulation to replicate natural sensationsreceived by the human body through the skin, muscles and bones.

Data from sensors on a remote UAV is used to indicate remote flightconditions via electrical stimulation (haptic cues) applied to the pilot(e.g., warning the pilot that the UV is in an unintended bankingcondition or is approaching an obstacle). The sensors of the UV aremapped to the operator so that the large muscle groups of the back,shoulders and thighs become indicators to the operator of the flightconditions of the UV. The applied signal causes a desired body positionrelated to a sensed parameter, such as flex, rotation, tilt, pitch, yaw,temperature, vibration, and other detectable stresses or conditions of amechanical component (wing, fuselage, control surfaces, etc.) of theUVS. The sensed parameter could be air pressure experienced at a wingcontrol surface while maneuvering. The sensed parameter is transmittedfrom the UV, causing a computer-controlled neuromuscular cues(electrical stimulation) resulting in an auto-action response in thehand of the pilot feeling pressure to assume a position directly relatedto the UV's control surface. The pressure to move the hand is the resultof muscle movements caused by the electrical stimulation. The pilotexperiences the sensation of resistance or pressure because of thecomputer-controlled electrical signals applied to the pilot's ownsensory/muscular physiology. In addition to pressure and resistance, thephysical sensation of vibrations, knocks and even scratches can beperceived as the result of subcutaneous controlled electrical signalstimulation. The muscle movements are involuntarily and automatic. Thereare no mechanical force simulators involved.

The uses for the HHMI go beyond UVS. The HHMI opens new avenues inhuman/machine interaction and control, that also impacts areas ofaccelerated learning, physical training and rehabilitation. The abilityto identify muscle groups at a sufficient level of definition, and theability to apply electrical signals at a similar level, enables an HHMIsystem in which previously-known actions and muscle movements could bedeveloped for improved physical training and correction of physicalmotion. Muscle memory associated with nearly all kinds of humanactivities can be more quickly developed to learn, for example, amusical instrument or sport technique. For military applications, beyondthe robotics and drones, rapid muscle memory build up could enhance thetraining of soldiers in basic and advanced weapons. Additionally, newforms of safety restraints could be imagined in which the human user isprevented from taking an action that may result in injury or acatastrophic vehicle accident.

The invention claimed is:
 1. An apparatus, comprising a wearableelectronic garment; and at least one processor, and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus at least to perform: receiving time sequential data;generating a plurality of haptic sensory cues capable of being perceivedby a user; wherein the haptic sensory cues are received by the user ascomputer controlled serially generated electrical signals, and whereinthe electrical signals invoke at least one of an involuntary musclecontraction and a perception by the user related to the sense of touch,and wherein the haptic sensory cues are generated in synchronizationdependent on the time sequential data; detecting activity from at leastone of muscles and nerves of a user; determining control intentions ofthe user dependent on the detected activity; determining an electricalsignal having signal characteristics based on the detected activity; andapplying the determined electrical signal to control an object dependenton the determined control intentions of the user, wherein the wearableelectronic garment has individually addressable electrodes for detectingthe activity as electrical activity from at least one of muscles andnerves of the user and for applying the computer controlled seriallygenerated electrical signals to invoke said at least one of theinvoluntary muscle contraction and the perception by the user related tothe sense of touch.
 2. The apparatus according to claim 1, furtherperforming, receiving time sequential first sensory data; and generatinga plurality of first sensory cues capable of being perceived by theuser, the plurality of first sensory cues being effective forstimulating at least one sense of the user; and wherein the hapticsensory cues are generated in synchronization dependent on the pluralityof first sensory cues.
 3. The apparatus according to claim 1, whereinthe time sequential data includes at least one sensed condition sensedremote in at least one of time, location and physical form from theuser.
 4. The apparatus according to claim 1, wherein the time sequentialdata is received from a remote transmitter associated with a remotelycontrolled vehicle; and wherein the object is the remotely controlledvehicle.
 5. The apparatus according to claim 1, further performing:generating a plurality of first sensory cues capable of being perceivedby a user, the plurality of first sensory cues being time sequentiallygenerated and effective for stimulating at least one sense of the user;and wherein the computer controlled serially generated electricalsignals invoke at least one of the involuntary muscle contraction andthe perception by the user related to the sense of touch as a perceptionby the user related to at least one of proprioception, mechanoreception,thermoception and nociception, and wherein the haptic sensory cues aregenerated in synchronization dependent on the time sequentiallygenerated plurality of first sensory cues.
 6. The apparatus according toclaim 5, wherein the plurality of first sensory cues comprises visualsensory cues for providing a virtual visual indication to the user of anevent.
 7. The apparatus according to claim 6, wherein the visual sensorycues are generated dependent on video data mapped to at least one ofcolor and intensity of an image of the event, and wherein the hapticsensory cues are generated dependent on the mapped video data to invokeat least one of the involuntary muscle contraction and the perception bythe user related to the sense of touch as having at least one of anintensity and invoked sensation dependent on the visual sensory cues. 8.The apparatus according to claim 5, wherein the plurality of firstsensory cues comprises auditory sensory cues for providing a virtualauditory indication to the user of an event.
 9. The apparatus accordingto claim 8, wherein the auditory sensory cues are generated dependent onsound data mapped to stereo, multichannel and/or binaural channels; andwherein the haptic sensory cues are generated dependent on the mappedsound data to invoke at least one of the involuntary muscle contractionand the perception by the user related to the sense of touch having atleast one of an intensity and invoked sensation dependent on theauditory sensory cues.
 10. The apparatus according to claim 1; whereinthe time sequential data is dependent on a beat from a piece of music.11. The apparatus according to claim 2, wherein the plurality of firstsensory cues comprises visual sensory cues for providing a virtualvisual indication to the user of an event, and wherein the visualsensory cues are generated dependent on video data mapped to at leastone of color and intensity of an image of the event, and wherein thehaptic sensory cues are generated dependent on the mapped video data toinvoke at least one of the involuntary muscle contraction and theperception by the user related to the sense of touch having at least oneof an intensity and invoked sensation dependent on the visual sensorycues.
 12. The apparatus according to claim 2, wherein the plurality offirst sensory cues comprises auditory sensory cues for providing avirtual auditory indication to the user of an event, and wherein theauditory sensory cues are generated dependent on sound data mapped to atleast one of intensity, frequency and location of a sound source of theevent, and wherein the haptic sensory cues are generated dependent onthe mapped sound data to invoke at least one of the involuntary musclecontraction and the perception by the user related to the sense of touchhaving at least one of an intensity and invoked sensation dependent onthe auditory sensory cues.
 13. The apparatus according to claim 12,further comprising generating a plurality of second sensory cues capableof being perceived by the user, the plurality of second sensory cuesbeing effective for stimulating a visual sense of the user; and whereinthe haptic sensory cues and the plurality of first sensory cues aregenerated in synchronization dependent on the plurality of secondsensory cues; and wherein the visual sensory cues are generateddependent on the mapped sound data to invoke visual stimulation havingcolors corresponding to the mapped sound data.
 14. The apparatusaccording to claim 2, further comprising generating a plurality ofsecond sensory cues capable of being perceived by the user, theplurality of second sensory cue being effective for stimulating at leastone of a visual and auditory sense of the user; and wherein the hapticsensory cues and the plurality of first sensory cues are generated insynchronization dependent on the plurality of second sensory cues. 15.The apparatus according to claim 1, wherein the object comprises one ofa biological component of the user, another user, or a remotely locatedmachine.
 16. The apparatus according to claim 1, wherein the detectedactivity from the at least one of muscles and nerves of the userincludes electromyography data; and further comprising at least one ofusing the detected electromyography data to record a performance of theuser, using analysis of the detected electromyography data to determineif the user is applying at least one of a learned skill and behavior,and using analysis of the detected electromyography data to providecognitive and physical therapy to the user.
 17. An apparatus, comprisingat least one processor, and at least one memory including computerprogram code, the at least one memory and the computer program codeconfigured to, with the at least one processor, cause the apparatus atleast to perform: detecting electrical activity from at least one ofmuscles and nerves of a user; determining control intentions of the userdependent on the detected electrical activity; determining an electricalsignal having signal characteristics based on the detected electricalactivity; and applying the determined electrical signal to control abody part of the user dependent on the determined control intentions ofthe user, wherein the applied electrical signal invokes at least oneinvoluntary muscle contraction in the body part of the user to controlthe body part dependent on the control intentions of the user.
 18. Theapparatus according to claim 17, further performing, generating aplurality of haptic sensory cues capable of being perceived by a user;wherein the haptic sensory cues are received by the user as computercontrolled electrical signals, and wherein the electrical signals invokeat least one of an involuntary muscle contraction and a perception bythe user related to the sense of touch.
 19. The apparatus according toclaim 18, further performing, generating a plurality of first sensorycues capable of being perceived by the user, the plurality of firstsensory cues being effective for stimulating at least one sense of theuser; and wherein the haptic sensory cues are generated insynchronization dependent on the plurality of first sensory cues. 20.The apparatus according to claim 19, wherein the plurality of firstsensory cues comprises visual sensory cues for providing a virtualvisual indication to the user of an event, wherein the visual sensorycues include video data mapped to at least one of color and intensity ofan image of the event, and wherein the haptic sensory cues are generateddependent on the mapped video data to invoke at least one of theinvoluntary muscle contraction and the perception by the user related tothe sense of touch as having at least one of an intensity and invokedsensation dependent on the visual sensory cues.
 21. The apparatusaccording to claim 19, wherein the plurality of first sensory cuescomprises auditory sensory cues for providing a virtual auditoryindication to the user of an event.
 22. The apparatus according to claim21, wherein the auditory sensory cues include sound data mapped tostereo, multichannel and/or binaural channels; and wherein the hapticsensory cues are generated dependent on the mapped sound data to invokeat least one of the involuntary muscle contraction and the perception bythe user related to the sense of touch having at least one of anintensity and invoked sensation dependent on the auditory sensory cues.23. The apparatus according to claim 18, further performing, receivingrime sequential data, wherein the time sequential data includes at leastone sensed condition sensed remote in at least one of time, location andphysical form from the user, and wherein the haptic sensory cues aregenerated in synchronization dependent on the time sequential data. 24.The apparatus according to claim 23, wherein the time sequential data isreceived from a remote transmitter associated with a remotely controlledvehicle.
 25. The apparatus according to claim 18: further performingreceiving time sequential first sensory data; and wherein the computercontrolled electrical signals invoke at least one of the involuntarymuscle contraction and the perception by the user related to the senseof touch as a perception by the user related to at least one ofproprioception, mechanoreception, thermoception and nociception, andwherein the haptic sensory cues are generated in synchronizationdependent on the time sequential first sensory data.
 26. The apparatusaccording to claim 18, further performing, generating a plurality offirst sensory cues comprising visual sensory cues for providing avirtual visual indication to the user of an event, and wherein thevisual sensory cues are generated dependent on video data mapped to atleast one of color and intensity of an image of the event, and whereinthe haptic sensory cues are generated dependent on the mapped video datato invoke at least one of the involuntary muscle contraction and theperception by the user related to the sense of touch having at least oneof an intensity and invoked sensation dependent on the visual sensorycues.
 27. The apparatus according to claim 18, performing, generating aplurality of first sensory cues comprising auditory sensory cues forproviding a virtual auditory indication to the user of an event, andwherein the auditory sensory cues are generated dependent on sound datamapped to at least one of intensity, frequency and location of a soundsource of the event, and wherein the haptic sensory cues are generateddependent on the mapped sound data to invoke at least one of theinvoluntary muscle contraction and the perception by the user related tothe sense of touch having at least one of an intensity and invokedsensation dependent on the visual auditory cues.
 28. The apparatusaccording to claim 27, further comprising generating a plurality ofsecond sensory cues effective for stimulating a visual sense of theuser; and wherein the haptic sensory cues and the plurality of firstsensory cues are generated in synchronization dependent on the pluralityof second sensory cues; and wherein the second sensory cues aregenerated dependent on the mapped sound data to invoke visualstimulation having colors corresponding to the mapped sound data. 29.The apparatus according to claim 17, wherein the detected electricalactivity from the at least one of muscles and nerves of the userincludes electromyography data; and further comprising at least one ofusing the detected electromyography data to record a performance of theuser, using analysis of the detected electromyography data to determineif the user is applying at least one of a learned skill and behavior,and using analysis of the detected electromyography data to providecognitive and physical therapy to the user.
 30. The apparatus accordingto claim 17, wherein the apparatus is configured as a wearableelectronic garment having individually addressable electrodes fordetecting the activity as electrical activity from at least one ofmuscles and nerves of the user and for applying the determinedelectrical signal to the user to invoke said at least one of involuntarymuscle contraction dependent on the control intentions of the user. 31.An apparatus, comprising at least one processor, and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus at least to perform: receiving time sequential data,wherein the time sequential data is dependent on a beat from a piece ofmusic; generating a plurality of haptic sensory cues capable of beingperceived by a user, wherein the haptic sensory cues are received by auser as computer controlled serially generated electrical signals, andwherein the electrical signals invoke at least one of an involuntarymuscle contraction and a perception by the user related to the sense oftouch, and wherein the haptic sensory cues are generated insynchronization dependent on the time sequential data; detectingactivity from at least one of muscles and nerves of at the user;determining control intentions of the user dependent on the detectedactivity; determining an electrical signal having signal characteristicsbased on the detected activity; and applying the determined electricalsignal to the legs of the user to control the legs of the user dependenton the determined control intentions of the user for physical andcognitive therapy to mitigate gait disturbance.
 32. The apparatusaccording to claim 31, wherein the detected electrical activity from theat least one of muscles and nerves of the user includes electromyographydata; and further comprising at least one of using the detectedelectromyography data to record a performance of the user, usinganalysis of the detected electromyography data to determine if the useris applying at least one of a learned skill and behavior, and usinganalysis of the detected electromyography data to provide cognitive andphysical therapy to the user.
 33. The apparatus according to claim 31,further performing, receiving time sequential first sensory data; andgenerating a plurality of first sensory cues capable of being perceivedby the user, the plurality of first sensory cues being effective forstimulating at least one sense of the user, and wherein the hapticsensory cues are generated in synchronization dependent on the pluralityof first sensory cues.
 34. The apparatus according to claim 33, whereinthe computer controlled serially generated electrical signals invoke atleast one of the involuntary muscle contraction and the perception bythe user related to the sense of touch as a perception by the userrelated to at least one of proprioception, mechanoreception,thermoception and nociception.
 35. The apparatus according to claim 33,wherein the plurality of first sensory cues comprises visual sensorycues for providing a virtual visual indication to the user of an event,where the visual sensory cues include video data mapped to at least oneof color and intensity of an image of the event, and wherein the hapticsensory cues are generated dependent on the mapped video data to invokeat least one of the involuntary muscle contraction and the perception bythe user related to the sense of touch as having at least one of anintensity and invoked sensation dependent on the visual sensory cues.36. The apparatus according to claim 31, wherein the apparatus isconfigured as a wearable electronic garment having individuallyaddressable electrodes for detecting the activity as electrical activityfrom at least one of muscles and nerves of a user and for applying theelectrical signals to the user to invoke said at least one of theinvoluntary muscle contraction and the perception by the user related tothe sense of touch.
 37. An apparatus, comprising at least one processor,and at least one memory including computer program code, the at leastone memory and the computer program code configured to, with the atleast one processor, cause the apparatus at least to perform: receivingtime sequential data; receiving time sequential first sensory data;generating a plurality of first sensory cues capable of being perceivedby a user, the plurality of first sensory cues being effective forstimulating at least one sense of the user, wherein the plurality offirst sensory cues includes auditory sensory cues for providing avirtual auditory indication to the user of an event, and wherein theauditory sensory cues are generated dependent on sound data mapped to atleast one of intensity, frequency and location of a sound source of theevent; generating a plurality of second sensory cues capable of beingperceived by the user, the plurality of second sensory cues beingeffective for stimulating a visual sense of the user; generating aplurality of haptic sensory cues capable of being perceived by a user;wherein the haptic sensory cues are received by the user as computercontrolled serially generated electrical signals, and wherein theelectrical signals invoke at least one of an involuntary musclecontraction and a perception by the user related to the sense of touch,and wherein the haptic sensory cues are generated in synchronizationdependent on the time sequential data and the mapped sound data toinvoke at least one of an involuntary muscle contraction and aperception by the user related to the sense of touch having at least oneof an intensity and invoked sensation dependent on the visual auditorycues, and wherein the haptic sensory cues and the plurality of firstsensory cues are generated in synchronization dependent on the pluralityof second sensory cues, and wherein the second sensory cues aregenerated dependent on the mapped sound data to invoke visualstimulation having colors corresponding to the mapped sound data;detecting activity from at least one of muscles and nerves of a user;determining control intentions of the user dependent on the detectedactivity; determining an electrical signal having signal characteristicsbased on the detected activity; and applying the determined electricalsignal to control an object dependent on the determined controlintentions of the user.
 38. The apparatus according to claim 37, whereinthe detected activity from the at least one of muscles and nerves of theuser includes electromyography data; and further comprising at least oneof using the detected electromyography data to record a performance ofthe user, using analysis of the detected electromyography data todetermine if the user is applying at least one of a learned skill andbehavior, and using analysis of the detected electromyography data toprovide cognitive and physical therapy to the user.
 39. The apparatusaccording to claim 37, wherein the time sequential data is received froma remote transmitter associated with a remotely controlled vehicle; andwherein the object is the remotely controlled vehicle.
 40. The apparatusaccording to claim 37, wherein the object comprises one of a biologicalcomponent of the user, another user, or a remotely located machine. 41.The apparatus according to claim 37, wherein the apparatus is configuredas a wearable electronic garment having individually addressableelectrodes for detecting the activity as electrical activity from atleast one of muscles and nerves of a user and for applying theelectrical signals to the user to invoke said at least one of theinvoluntary muscle contraction and the perception by the user related tothe sense of touch.
 42. An apparatus, comprising at least one processor,and at least one memory including computer program code, the at leastone memory and the computer program code configured to, with the atleast one processor, cause the apparatus at least to perform: receivingtime sequential data; generating a plurality of first sensory cuescapable of being perceived by the user, the plurality of first sensorycues being effective for stimulating at least one sense of the user;generating a plurality of second sensory cues capable of being perceivedby the user, the plurality of second sensory cue being effective forstimulating at least one of a visual and auditory sense of the user;generating a plurality of haptic sensory cues capable of being perceivedby a user; wherein the haptic sensory cues are received by the user ascomputer controlled serially generated electrical signals, and whereinthe electrical signals invoke at least one of an involuntary musclecontraction and a perception by the user related to the sense of touch,and wherein the haptic sensory cues are generated in synchronizationdependent on the time sequential data and the plurality of first sensorycues; and wherein the haptic sensory cues and the plurality of firstsensory cues are generated in synchronization dependent on the pluralityof second sensory cues; detecting activity from at least one of musclesand nerves of a user; determining control intentions of the userdependent on the detected activity; determining an electrical signalhaving signal characteristics based on the detected activity; andapplying the determined electrical signal to control an object dependenton the determined control intentions of the user.
 43. The apparatusaccording to claim 42, wherein the time sequential data includes atleast one sensed condition sensed remote in at least one of time,location and physical form from the user.
 44. The apparatus according toclaim 42, wherein the time sequential data is received from a remotetransmitter associated with a remotely controlled vehicle; and whereinthe object is the remotely controlled vehicle.
 45. The apparatusaccording to claim 42, wherein the detected activity from the at leastone of muscles and nerves of the user includes electromyography data;and further comprising at least one of using the detectedelectromyography data to record a performance of the user, usinganalysis of the detected electromyography data to determine if the useris applying at least one of a learned skill and behavior, and usinganalysis of the detected electromyography data to provide cognitive andphysical therapy to the user.
 46. The apparatus according to claim 45,wherein the plurality of first sensory cues comprises visual sensorycues for providing a virtual visual indication to the user of an event.47. The apparatus according to claim 46, wherein the visual sensory cuesinclude video data dependent on visual images of the event, and whereinthe haptic sensory cues are generated dependent on the visual sensorycues to invoke at least one of the involuntary muscle contraction andthe perception by the user related to the sense of touch dependent onthe visual images of the event.
 48. The apparatus according to claim 45,wherein the plurality of first sensory cues comprises auditory sensorycues for providing a virtual auditory indication to the user of anevent.
 49. The apparatus according to claim 48, wherein the auditorysensory cues are generated dependent on sound data of the event; andwherein the haptic sensory cues are generated dependent on the mappedsound data to invoke at least one of the involuntary muscle contractionand the perception by the user related to the sense of touch dependenton the sound data.
 50. The apparatus according to claim 42, wherein theplurality of first sensory cues comprises visual sensory cues forproviding a virtual visual indication to the user of an event, andwherein the visual sensory cues are generated dependent on video datamapped to at least one of color and intensity of an image of the event,and wherein the haptic sensory cues are generated dependent on themapped video data to invoke at least one of the involuntary musclecontraction and the perception by the user related to the sense of touchhaving at least one of an intensity and invoked sensation dependent onthe visual sensory cues.
 51. The apparatus according to claim 42,wherein the plurality of first sensory cues comprises auditory sensorycues for providing a virtual auditory indication to the user of anevent, and wherein the auditory sensory cues are generated dependent onsound data mapped to at least one of intensity, frequency and locationof a sound source of the event, and wherein the haptic sensory cues aregenerated dependent on the mapped sound data to invoke at least one ofthe involuntary muscle contraction and the perception by the userrelated to the sense of touch having at least one of an intensity andinvoked sensation dependent on the visual auditory cues.
 52. Theapparatus according to claim 42, wherein the object comprises one of abiological component of the user, another user, or a remotely locatedmachine.
 53. The apparatus according to claim 42, wherein the timesequential data comprises time series information that is at least oneof generated, recorded, transmitted and received as at least one of ananalog and a digital signal.
 54. The apparatus according to claim 42,wherein the apparatus is configured as a wearable electronic garmenthaving individually addressable electrodes for detecting the activity aselectrical activity from at least one of muscles and nerves of a userand for applying the determined electrical signal to the user to invokesaid at least one of the involuntary muscle contraction and theperception by the user related to the sense of touch.